Repository navigation
Docs: how can new users try creating a dataset? #129
Description
Hello, and thank you for merging the documentation update #126. I tested the getting-started tutorial as a new user to see what I could do next. Reading a dataset works, but I cannot complete the dataset creation part.
I created an API token on data.humdata.org, as described in "Obtain your API Key", and used it with hdx_read_only=False. Step 9 of the tutorial uses hdx_site="stage" for writing:
from hdx.api.configuration import Configuration
from hdx.data.hdxobject import HDXError
from hdx.data.user import User
Configuration.create(hdx_site="stage", user_agent="stage_access_repro")
try:
User.get_current_user()
except HDXError as e:
print(e)
print(repr(e.__cause__))Output:
Failed when trying to show! (POST)
NotAuthorized({'__type': 'Authorization Error', 'message': 'Access denied'})
The same code with hdx_site="prod" authenticates (User.get_current_user() returns my username), which seems to prove that the issue comes from the stage site. That account has no organisation, so it cannot create datasets on prod, which matches the message on the "Add Data" page from data.humdata.org. I believe that is intended.
The demo site (for example https://demo.data-humdata-org.ahconu.org/, which the README uses for test_ckan.py) returns HTTP 401 for a plain request, so I assume outside users can't use them. Because of this, the tutorial's write steps can't be followed by an outside reader.
Questions:
- What should an outside reader use to practise creating and updating a dataset: a public test site, a sandbox organisation, or something else?
- If there is no such option, would you be happy with a tutorial that builds and validates the dataset locally and shows create_in_hdx() as the step for organisation members?
- Who do you see as the main readers of the docs (publishers inside organisations, data analysts reading data, or both)? That would help me choose what to write first.
I am happy to write the tutorial and a "Troubleshoot common errors" how-to page, but I want them to match what you would actually point new users to.
Best,
Antonin
Activity
mcarans commentedon Nov 5, 2026
Thanks for this issue which highlights gaps in our documentation. Although we have additional documentation here: https://docs.humdata.org/build/hdx-python-library, it does not cover the authorisation issue you have raised.
Our intention is that people should use stage or feature if they want to create datasets that are unlikely to be touched for a while. The issue here is that the stage or feature) database is periodically (rarely) updated from prod so the the API key (or anything else created on prod) won't be on stage if the update has not occurred. The best way to do it is to be prepared to create a separate API key on stage for testing. demo is updated weekly so any changes on it that have not come from prod will be lost (but conversely prod API keys and other changes will be available within at most a week).
I'm curious still about this tutorial: how would it work without a test site?
I would say both
@AntoninKns Please tell me more about the tutorial you have in mind.
AntoninKns commentedon Nov 11, 2026
Thank you @mcarans for the answer, the explanation, and for the link to the HDX docs. I am sorry, I had not seen that page when I opened the issue.
What I found when retesting
For someone who isn't in an organisation, reading works everywhere but creating a dataset doesn't, and I don't see a way around that. Is there a sandbox organisation or another way for a new user to get publishing rights on a test server, or is it always done by request to hdx@un.org like in the link you sent?
To your question about how the tutorial would work without a test site. I put everything in the draft PR: #131 to make it easier to visualize. Below is a summary of it. I separated the "Getting started" tutorial into "Read a dataset" and "Create a dataset". I believe it could make scope clearer and avoid needed API tokens for people that only want to read the data.
The main part of the "create a dataset" tutorial runs offline. The reader starts from a CSV, builds the dataset, generates the resource, validates it with check_required_fields() and looks at what would be sent, using only a read-only connection. It stops just before create_in_hdx() and explains the error a non-member would get. A last section, for organisation members, explains how to rehearse on demo or stage (including your point about needing a separate key on stage) and then publish to prod. I couldn't run that section, so it says so.
Thank you again for following up on those and sorry for my delayed answer, I figured answering with the PR would be easier to visualize.
Best,
Antonin