[jimdowling]

Summary

• New guide user_guides/fs/data_source/creation/sharepoint.md: the Entra ID application and its permissions, the cluster variables, creating a data source and a sync in the UI, the data source page, how a sync runs (job, editable script, read-only dataset, CSV audit log with a DuckDB example, failures, deletion), and the Python client and CLI. Added to the nav.
• The strict build passes for this page; the build's remaining warnings are agent pages that need a newer hopsworks-api than this ticket's branch.

Related PRs (merge in this order)

1. logicalclocks/docker-images#1138
2. logicalclocks/hopsworks-ee#3573
3. logicalclocks/hopsworks-helm#2627
4. [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset hopsworks-api#1353
5. logicalclocks/hopsworks-front#2201
6. logicalclocks/loadtest#1186
7. [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset #715

Feature

HWORKS-3562: a SharePoint Online data source that copies a folder of a site's document library into a project dataset, once or on a schedule (one-way mirror). Each sync is an ordinary Python job in the python-feature-pipeline environment running DataSources/<connector>/sharepoint_sync.py (rclone), which data owners can edit. Every run that changes something writes a CSV audit file in Hive-style date= partitions, rotated after data_source_audit_retention_days. Failure alerts go to an optional alert receiver. Two-way sync and delegated (user) sign-in are out of scope.

Merge order: docker-images (image with rclone and duckdb) → hopsworks-ee → hopsworks-helm → hopsworks-api → hopsworks-front → loadtest → docs.

Testing

On cluster jim-sharepoint (5.3.0-SNAPSHOT) against a real Microsoft 365 tenant (site hopsworks-sync-test, app with Sites.ReadWrite.All), with the backend, frontend and SDK from these branches and the python-feature-pipeline image built from the docker-images branch:

• loadtest workflows/feature_store/python_driver/test_data_sources_sharepoint.py: 2 passed in 4m13s (2026-10-24, after merging upstream's SQL Server and Hopsworks data sources; SHAREPOINT is ordinal 20, after HOPSWORKS)
  • copy once: copies a folder subtree, audits added files and folders, a second run after a deletion in SharePoint deletes nothing;
  • scheduled mirror: initial copy, an idle run writes no audit file, then update, rename, delete and add in SharePoint are mirrored with matching audit rows; deleting the synced dataset is refused (110057) while a data owner may delete a file in it, and the next run restores it; pause, then delete without keeping files removes the dataset.
• UI walk-through in a browser: data source form, test connection (toast), sync setup with the live library and folder tree, schedule presets, Syncs table and audit card with the DuckDB snippet, Code tab, file browser (sync icon, write actions hidden), Jobs badge, delete dialogs.
• Unit tests: ee SharePoint Java tests (31) and the sync script's pytest suite (36, with rclone 1.75.2 and duckdb 2.0.0 in the image); api SDK and CLI tests; front vitest for the storage-connector pages (81).
• Reviews: a Codex pre-review of all repos and three adversarial rounds on the lifecycle and fencing code; then a Codex review of these PRs (site-scoped libraries, update validation and alert replacement, pending deletes in the SDK and CLI, UI error and feedback states, guide corrections) and three more adversarial rounds on those fixes. Every finding was verified and fixed or recorded. Site scoping, rejected updates and the transfers reset were also checked live. Copilot could not review (quota).

Not verified on a cluster:

• the packaged Flyway migration and Helm seeding on an upgrade (the DDL and the nine variables were applied by hand on the test cluster);
• Sites.Selected with a site-level grant (the test app has Sites.ReadWrite.All);
• a scheduler-fired run (the loadtest starts runs explicitly; scheduled-run argument handling is unit-tested);
• delivery of failure alerts to a receiver (none configured on the test cluster).

🤖 Generated with Claude Code

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.