ISOC-CH/tech-dossier-project-data
3
0
Fork 0
Code Issues 2 Pull requests 2 Projects Releases Packages Wiki Activity Actions

WIP: feat(scripts): initial commit ingestor #4

Draft
puer-robustus wants to merge 2 commits from ingestor into main
pull from: ingestor
merge into: ISOC-CH:main
Conversation 2 Commits 2 Files changed 2 +112
puer-robustus commented 2026-07-06 20:19:01 +02:00
Owner
Copy link
No description provided.
feat(scripts): initial commit ingestor
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ci/woodpecker/push/woodpecker Pipeline failed
Details
1d11f24375
puer-robustus commented 2026-07-06 20:23:03 +02:00
Author
Owner
Copy link

@hernani @ijohn For your review.

Running the script (python3 scripts/ingest.py with the necessary python dependencies installed) will create a whole bunch of json files in data/projects which look like this:

{
    "name": "crypto.signText",
    "website": [
        "https://redwax.eu/rst"
    ],
    "description": "Cryptographic signatures brought back to the browser",
    "ngiFund": [
        "ngi-assure"
    ]
}

Now, this in and off itself is not much, as it is just reformatting the ngi grant level data. However, the script has the capabilities to aggregate project level data from multiple grants, IF the variables GRANTID2PROJECT and PROJECT2DESCRIPTION are correctly filled in. The question now is, how do we fill in these data programmatically? @ijohn Is there a way to get the hex data in a more consumable form than parsing the html?

@hernani @ijohn For your review. Running the script (`python3 scripts/ingest.py` with the necessary python dependencies installed) will create a whole bunch of json files in `data/projects` which look like this: ``` { "name": "crypto.signText", "website": [ "https://redwax.eu/rst" ], "description": "Cryptographic signatures brought back to the browser", "ngiFund": [ "ngi-assure" ] } ``` Now, this in and off itself is not much, as it is just reformatting the ngi grant level data. However, the script has the capabilities to aggregate project level data from multiple grants, **IF** the variables `GRANTID2PROJECT` and `PROJECT2DESCRIPTION` are correctly filled in. The question now is, how do we fill in these data programmatically? @ijohn Is there a way to get the hex data in a more consumable form than parsing the html?
style(scripts/ingest.py): add REUSE header
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ci/woodpecker/push/woodpecker Pipeline failed
Details
0fbfd1f961
puer-robustus commented 2026-07-06 20:26:24 +02:00
Author
Owner
Copy link

The resulting json files are also not schema compliant yet.

The resulting json files are also not schema compliant yet.
puer-robustus force-pushed ingestor from 0fbfd1f961
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ci/woodpecker/push/woodpecker Pipeline failed
Details
to a6569922d6
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ci/woodpecker/push/woodpecker Pipeline failed
Details
2026-07-06 20:29:26 +02:00 Compare
Some checks failed
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ci/woodpecker/push/woodpecker Pipeline failed
Details
This pull request is marked as a work in progress.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin ingestor:ingestor
git switch ingestor

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff ingestor
git switch ingestor
git rebase main
git switch main
git merge --ff-only ingestor
git switch ingestor
git rebase main
git switch main
git merge --no-ff ingestor
git switch main
git merge --squash ingestor
git switch main
git merge --ff-only ingestor
git switch main
git merge ingestor
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
ISOC-CH/tech-dossier-project-data!4
No description provided.