{ "version": "https://jsonfeed.org/version/1", "title": "Stephen A. Fuqua Blog", "home_page_url": "https://blog.safnet.com/", "description": "Stephen A. Fuqua writes about technology, the natural world, and the Bahá'í Faith", "items": [ { "id": "https://blog.safnet.com/2026/09/07/sandboxes", "content_html": "

TL;DR summary: Docker Sandboxes are a convenient and effective means of safely running a coding harness in permissions bypass mode.

\n

YOLO is now (dangerously so) the norm​

\n

Running coding harnesses in YOLO (bypass permissions) mode has become the new norm:

\n\n

In YOLO mode, the coding harnesses in summer 2026 seem to be very good at preventing the agents from reaching files outside the code repository; in all of 2026 I do not think I have denied permission to anything, and my spot-checking of the logs revealed nothing concerning.

\n

Then again, they might just be very good at hiding their tracks... see recent news about agents \"escaping\" to crack into other companies.

\n

Probing the limits of permissions and worktrees​

\n

Assume for a moment that the agent won't do anything to trash your computer. What about production systems? Across dozens of repositories, I have two .env files with production system information. Attempting to deny access to read those files is tricky, at least with Claude Code: the system can often find a way using indirect access through an approved rule. For example, denying \"Read(./.env.*)\" will work, but will that prevent it from calling cat .env from a Bash command? And did you remember to restore access to a file like .env.example so that the agent can insert new environment variables?

\n

\"This

\n

Running the agent on a git worktree will help: the worktree won't have the environment files. But what about environment variables? Or signed-in tools? Do you want the tool to run your cloud provider's CLI tool on your behalf to check on production settings?

\n

Getting smart​

\n

I have a personal project, building a web application for volunteer management. The deployment is set up on Azure. A few weeks ago, I tasked Claude Code with inspecting deployment scripts to try to diagnose and correct a production error.

\n

Guiltily, I recognized my guard was down: were there previous occasions in which Claude Code, in bypass mode, was accessing my Azure subscription without adequate safeguards? That was a wake up call.

\n

Requirements​

\n

Goal: select an approach for safely running in YOLO mode, satisfying this non-exhaustive list of requirements:

\n\n

A quick survey of solutions​

\n

Simplistic thinking: git worktree and WSL. Let's test that theory:

\n
$ az account list
[
  {
    \"cloudName\": \"AzureCloud\",
    \"homeTenantId\": \"*****\",
    \"id\": \"*****\",
    \"isDefault\": true,
    \"name\": \"*****\",
    \"state\": \"Enabled\",
    \"tenantDefaultDomain\": \"*****.onmicrosoft.com\",
    \"tenantDisplayName\": \"Default Directory\",
    \"tenantId\": \"\"*****\",
    \"user\": {
      \"name\": \"*****\",
      \"type\": \"user\"
    }
  }
]
\n

Nope, az is accessing the same session authentication as seen from Windows (outside of WSL).

\n

Next: Claude Code has a number of ideas for sandboxing.

\n
ApproachMeets Requirements
Sandboxed Bash❌ no - Claude specific
Sandbox runtime❌ no - Claude specific
Dev container✅ looks intriguing
Custom container✅ let's try this
Virtual machine❌ no - too intensive
Claude Code on the web⚠️ yes - but see below
\n
tip

Assigning work to Claude Code on the web, or GitHub Copilot Coding Agent, can be a productive and safe experience. In fact, I delegate most final implementations to remote agents. But I prefer doing my planning locally; my planning artifacts are often at the wrong level of detail for long-term storage, so I prefer never to add them to the repository, as would occur if working only online.

The repository should have requirements (PRD) and architectural designs (ADRs and related documents), not low-level or interim implementation planning documents. The implementation plan that derives from a planning session is what I hand off to the remote agent.

\n

What about dev containers? Has promise, but I've yet to use a dev container in any setting: there will be a learning curve. Not rejected, but not yet ready to commit to it.

\n

And custom containers? I am very used to those. To jumpstart, a search: \"use docker as a sandbox for claude code\".

\n

Oh! Docker Sandboxes are a thing. A product. Pre-configured and free. What's the catch? You must sign-in; corporate governance tools are available but require a paid account. Out of the box, it meets all requirements, though #5 (Must be able to install dev tools) will require a little effort. Why build my own custom container when I can use this product?

\n

Solution​

\n

Two weeks later: I won't say this is the best solution. Having passed on trying out other options, that claim would be disingenuous. But it is working for me.

\n

As a regular Docker user, the command line interface is familiar and easy to learn. The containers run with a small footprint micro VM architecture, and the solution is completely separate from Docker Desktop (which is not required).

\n

At last, I feel that I'm practicing safe permission bypass. That is, assuming the agent doesn't do anything crazy with my GitHub token 😉.

\n

Tips​

\n\n
warning

Separate sandbox containers for each harness, spanning dozens of repositories, could eventually eat a lot of disk space. Will need to keep an eye on that going forward.

\n

Postscript: Future Direction​

\n

While this is working well, eventually I may get frustrated at some quirks to this approach. Perhaps I will come around to building my own container containing the compiler / runtime tools I need, along with both coding harnesses that I actively use. Which starts to sound a lot like a devcontainer...

\n
\n

This blog post was handcrafted and conceived, with minor only minor updates from Copilot Review Agent and a Claude Code + Sonnet 5 editorial review.

", "url": "https://blog.safnet.com/2026/09/07/sandboxes", "title": "Sandboxing for YOLO Safety", "summary": "TL;DR summary: Docker Sandboxes are a convenient and effective means of safely running a coding harness in permissions bypass mode.", "date_modified": "2026-09-07T00:00:00.000Z", "tags": [ "programming", "technology" ] }, { "id": "https://blog.safnet.com/2026/08/13/agentic-code-review", "content_html": "

From time to time I pop in to peek at one of the dozens of pull requests that come through my mailbox daily, furthering the work of the Ed-Fi Alliance to help connect student data. What feature, improvement, or bug fix are the teams working on? How has this engineer approached a given problem, what tests have they written, what documentation have they provided, are the I/O seams sufficiently fault tolerant? Did they run the GitHub Copilot Code Review Agent, and fix the issues detected there?

\n
tip

TL;DR Created a team of specialist reviewers skill and use it on anything non-trivial. After its analysis, it offers to post a nicely-formatted review comment on the PR. I then go back and edit that review comments, to avoid slop and/or excessive cognitive load.

\n

As a product manager — barely a part-time engineer — there is little time for this. And there is a lot to review, given that agentic coding work shifts the bottleneck from coding to review and test. So here too I have tried to use my coding harnesses well.

\n

The Copilot code review helps, but it is insufficient. I see it as a minimum bar: once that review has been cleared, the pull request (PR) is ready for another look. I scan with my own eyes, sampling bits of code to see the null and exception handling, log messages (and levels), and database connectivity. Only then (ideally) do I ask my coding harness (Claude Code, GitHub Copilot, etc.) to review the PR.

\n

It started simply: switch to the repo's local directory, open coding harness, and prompt \"review PR 123 and post your comments back to the PR.\" So far so good. Run it again: different results each time. OK, how can I be a bit more thorough? Specialization and subagents: \"review PR 123 with parallel agents covering security, coding quality, and maintainability. Summarize the subagents and report results back to me.\"

\n

After a few times of this, it had grown to five agents: security, functionality (with a Jira ticket to read), maintainability, usability, and test coverage. Five — if I remembered to use all of them. And how do we remember to use the same instructions? A skill! So I turned to the Superpowers skill-building skill to capture more detailed notes on my Team of Specialists, creating: team-pr-review.

\n
---
description: Launches a fleet of five specialist subagents to deeply review a pull request across security, functionality, maintainability, usability, and test coverage. Provide the PR number and related Issue number to get a severity-ranked summary of all findings.
name: team-pr-review
---
\n

The harness and I had a conversation to fill in additional repeatable detail. It came up with a summary format to my liking. We added yet more detail on each specialist. I refined it a few times, most notably telling it to post the comment using a markdown file (gh pr comment -F <file>) instead of posting the comment inline (gh pr comment -b \"<comment>\") — this resolved an issue where the harness kept escaping the backtick character incorrectly, leading to slightly malformed comments.

\n

And this is working well. But it risks throwing slop back at the programmer.

\n

There are typically 10+ review comments, especially on AI-produced commits that were very good, but not quite perfect. Occasionally I disagree with the severity or even the finding. So I began editing the review, after posting it. I don't want to waste tokens on the edits, so I do this by hand. I move items around in severity assessment, delete them or offer additional commentary, and now completely remove the \"nits\" section or bump those items up to low severity if I want to see them be resolved. And I add a note at the top mentioning that I have reviewed and edited the agent's review, so that the reader will know that I am not (one hopes!) forcing them to spend time analyzing something that I was unwilling to analyze myself.

\n

And if I'm really smart, I might ask the coding harness to look at the PR for updates to make to AGENTS.md. But be careful with that: you don't want to overwhelm that file.

\n
tip

Wait until you have performed several detailed reviews this way. Then start a coding session with something like this prompt:

\n

Review the past 5 pull requests and look for patterns in the review comments.

\n

When you find a repeated pattern, draft a proposed update to AGENTS.md. Do not immediately edit the file; present the recommendation to me with an offer to edit the file.

\n
\n
\n

AI usage disclosure: GitHub Copilot improved the title and provided a few editorial tips on writing clarity and grammar. In other words, the text is all mine.

", "url": "https://blog.safnet.com/2026/08/13/agentic-code-review", "title": "From Ad-Hoc Prompts to a Repeatable Agentic PR Review Workflow", "summary": "From time to time I pop in to peek at one of the dozens of pull requests that come through my mailbox daily, furthering the work of the Ed-Fi Alliance to help connect student data. What feature, improvement, or bug fix are the teams working on? How has this engineer approached a given problem, what tests have they written, what documentation have they provided, are the I/O seams sufficiently fault tolerant? Did they run the GitHub Copilot Code Review Agent, and fix the issues detected there?", "date_modified": "2026-08-13T00:00:00.000Z", "tags": [ "programming", "technology" ] }, { "id": "https://blog.safnet.com/2026/04/10/cli-ai-harness", "content_html": "

For all of last year, I wondered why anyone would use the CLI version of Claude Code and similar tools. I asked a colleague about this a couple of times, and still had trouble getting it.

\n

Until I sat down and used the CLI to perform my work. As a knowledge worker, if you haven't used copilot or claude or (favorite tool) in the terminal, you are truly missing out.

\n

🔝 Top 3 reasons I like this experience:

\n

\"Claude

\n
    \n
  1. VS Code chat panel puts my conversation in the bottom right corner in a tiny box (by default). You can change this - but I hadn't thought to change it. The terminal is front-and-center, wherever I want it.
    2. \n
  2. These CLI tools are powerful with amazingly rich interfaces.
    3. \n
  3. We should all be shifting from \"software engineer assisted by AI\" to \"software engineering assisting AI\". Call it \"agentic engineering\" or \"AI orchestrator\" or whatever. The CLI makes you focus on the conversation with the model, without the distraction of the file explorer or a big blank space where you would normally be typing into a file.
    4. \n
\n

As with anything, there is a learning curve. You can always ask the tool directly for help. And make sure you read the tool's documentation in order to get the most out of it.

\n
\n

These words above went viral on LinkedIn, much to my surprise.

\n

I had made a provocative statement about moving from \"AI assisting the engineer\" to \"engineer assisting the AI\". This is, of course, overly simplistic. The statement is intended to spur conversation. 🧔‍♂️🛟🤖

\n

A case in point: Claude Code developed a simple analytics report for me with queries against a Cosmos DB collection. I have never personally learned SQL syntax for Cosmos DB. While testing, I found several mistakes in the scripts. After the tool corrected several of those mistakes, I asked Claude (Sonnet 4.6 model) to review the syntax again for any further errors: all good! it said.

\n

Then I asked it \"is VALUE a reserved word?\" At which point it realized that yes, it had made a mistake, and it corrected it.

\n

My deep history of writing SQL queries allowed me to guide the AI tool to fix its mistake. I still did not write any code by hand here - I let the CLI tool handle it - though I did use my editor to open and read the file in question. Engineer assisting the AI.

", "url": "https://blog.safnet.com/2026/04/10/cli-ai-harness", "title": "Falling in Love with the CLI AI Harness", "summary": "For all of last year, I wondered why anyone would use the CLI version of Claude Code and similar tools. I asked a colleague about this a couple of times, and still had trouble getting it.", "date_modified": "2026-04-10T00:00:00.000Z", "tags": [ "programming", "technology" ] }, { "id": "https://blog.safnet.com/2025/12/30/inward-turn", "content_html": "

Bahá'u'lláh's teachings focus, at their core, on the concept of the essential unity of humanity: that we are biologically one \"species\", we should expand our notions of belonging and compassion to the entirety of humanity, and that doing so is the road to peace. [1]

\n

Another of His teachings is the concept of moderation [2]. One manifestation of that moderation is in the balancing act between \"units\" of humanity: individual, family, community, institutions, government. Each unit has its prerogatives [3], its spheres of action, overlapping one another, seeking to avoid infringement on the others whilst simultaneously protecting the harmony of all.

\n

I was stirred into contemplation of \"the individual\" within these overlapping contexts on reading the following quote from Karl Jaspers, in The Origin and Goal of History [4]:

\n
\n

\"Man proved capable of contrasting himself inwardly with the entire universe. He discovered within himself the origin from which to raise himself above his own self and the world.\"

\n
\n

Thus the \"axial age\" (roughly 8th to 2nd centuries BCE), as Jaspers calls it, saw the rise of individuals distinct from tribe and distinct from the rest of the universe (\"nature\"); unique, soul-embodied, self-reflecting, self-aware, and (potentially) self-absorbed. This inward turn led me back to a favorite quotation from Baha'u'llah (The Hidden Words, number 13 from the Arabic):

\n
\n

\"O SON OF SPIRIT!

\n

I created thee rich, why dost thou bring thyself down to poverty? Noble I made thee, wherewith dost thou abase thyself? Out of the essence of knowledge I gave thee being, why seekest thou enlightenment from anyone beside Me? Out of the clay of love I molded thee, how dost thou busy thyself with another? Turn thy sight unto thyself, that thou mayest find Me standing within thee, mighty, powerful and self-subsisting.\"

\n
\n

Among the rich possibilities for interpretation and application, this short passage induces feelings of individual worth (nobility) and awareness of two larger spiritual truths: the personal \"thou\" is addressed to each human separately and simultaneously, and we are profoundly connected by finding our spiritual grounding (God, if you will) as an intrinsic element of self, which does not require extrinsic imposition or validation. In other words, as individuals we can know not only our selves (\"know thyself\") but also God (whatever we might mean by that word).

\n

Thus we see unity − a balance and moderation − instead of paradox when considering that the mechanism that Jaspers argues \"raise[s] (man) above his own self and the world\" may also lead deeper into spiritual communion.

\n

Returning to the idea of moderation, and the balance between individual and collective, a moment's reflection brings to mind several unities of opposites, each worthy of future further consideration:

\n
IndividualCommunal
Investigation and acceptance of truthConsultation as search for truth
Obligatory prayersCommunal (non-congregational) worship
Interpretation and applicationAuthoritative interpretation and elucidation
\n
\n

[1] See, for example, `Abdu'l-Bahá's First Tablet to the Hague. The entire letter is on this theme, but see in particular:

\n
\n

\"O peoples of the world! The Sun of Truth hath risen to illumine the whole earth, and to spiritualize the community of man. Laudable are the results and the fruits thereof, abundant the holy evidences deriving from this grace. This is mercy unalloyed and purest bounty; it is light for the world and all its peoples; it is harmony and fellowship, and love and solidarity; indeed it is compassion and unity, and the end of foreignness; it is the being at one, in complete dignity and freedom, with all on earth.\"

\n
\n

[2] One of my favorite examples of moderation is in Gleanings from the Writings of Bahá'u'lláh, CLXIV: \"Whoso cleaveth to justice, can, under no circumstances, transgress the limits of moderation.\" The passage continues by describing the need for moderation in \"civilization\" (technological development).

\n

[3] 'Abdu'l-Bahá explains his views on the family during a talk, views which find their echo in other commentary on individuals, communities, and institutions:

\n
\n

\"According to the teachings of Bahá'u'lláh the family, being a human unit, must be educated according to the rules of sanctity. All the virtues must be taught the family. The integrity of the family bond must be constantly considered, and the rights of the individual members must not be transgressed. The rights of the son, the father, the mother—none of them must be transgressed, none of them must be arbitrary. Just as the son has certain obligations to his father, the father, likewise, has certain obligations to his son. The mother, the sister and other members of the household have their certain prerogatives. All these rights and prerogatives must be conserved, yet the unity of the family must be sustained. The injury of one shall be considered the injury of all; the comfort of each, the comfort of all; the honor of one, the honor of all.\"

\n
\n

[4] The quote from Jaspers came to my attention through the book Digital Future in the Rearview Mirror (p 176), by Andrey Mir.

", "url": "https://blog.safnet.com/2025/12/30/inward-turn", "title": "The Inward Turn", "summary": "Bahá'u'lláh's teachings focus, at their core, on the concept of the essential unity of humanity: that we are biologically one \"species\", we should expand our notions of belonging and compassion to the entirety of humanity, and that doing so is the road to peace. [1]", "date_modified": "2025-12-30T00:00:00.000Z", "tags": [ "discourse", "bahai-faith" ] }, { "id": "https://blog.safnet.com/2025/10/23/program-birth", "content_html": "

Today we are celebrating the second of the Twin Holy days in the Bahá'í\ncalendar: the birth of Bahá'u'lláh, founder of the Bahá'í Faith, in 1817.\nYesterday we celebrated the birth of the Báb, born in 1819, who as an\nindependent messenger from God both declared a new religion as a break from\nIslam, and prophesied that another, greater revelation would come soon after\nHis own. Both were born in Persia, which is today known as Iran. In the lunar\ncalendar used by the Muslim world, their births fell on consecutive days.\nConsequently, Bahá'ís around the world celebrate them as the \"Twin Holy Days\".

\n
\"Entrance

Entrance to the Bahjí Pilgrim House, near the Shrine of Bahá'u'lláh (source)

\n

Continue...

", "url": "https://blog.safnet.com/2025/10/23/program-birth", "title": "Commemorating the Birth of Bahá'u'lláh", "summary": "Today we are celebrating the second of the Twin Holy days in the Bahá'í", "date_modified": "2025-10-23T00:00:00.000Z", "tags": [ "bahai-faith" ] }, { "id": "https://blog.safnet.com/2025/06/17/copilot-commit-signing", "content_html": "

GitHub Copilot Coding\nAgent is\nan impressive and powerful tool for \"autonomously\" completing development tasks\nrunning in GitHub's environment. Like any tool, there is a learning curve ahead\nbefore any of us become truly high functioning, beginning with the statement of\na well crafted prompt.

\n

⚠️ But I have a concern: lack of commit signing. Should that stop us from adopting\nCoding Agent? TL;DR conclusion: time to relax the commit signing requirement.

\n

What is commit signing?​

\n

Git commit signing uses public/private key encryption to validate the identity\nof the individual who created a given commit. This approach may help reduce the\nrisk and opportunity for malicious actors to attack the software supply\nchain; without\nsigning, it is trivial to spoof the commit as coming from a trusted source.\nGitHub's Well-Architected\nFramework\nreflects on this: \"Initiate and impose commit signing whenever possible. This\nwill deter malicious actors from creating a commit with malicious code and help\nprevent a possible supply chain attack.\" My organization has adopted this\npractice across our repositories.

\n

Spoofing a commit​

\n

Take a look at this pull\nrequest I\ncreated.

\n

\"Fake

\n

Looks like Copilot created that commit, doesn't it? But I was the originator.\nNote the lack of a green \"VERIFIED\"\nbadge.\nAfter using Coding Agent on an issue in this repository, I fetched the\nrepository, looked closely at the committer in the ref log, and used that info\nto spoof the author of my new commit.

\n

GitHub Copilot Agent Mode's limitation​

\n

I had to disable the signed commit requirement before I could accept the Coding\nAgent pull request. There is no way (that I can find) of configuring Coding\nAgent to sign commits - even though other bots, such as Dependabot, are able to\ndo this. And there is no way to set a branch rule that lets Copilot bypass the\ncommit signing requirement.

\n

What do others do? Microsoft, for example, does not require signed\ncommits. The final merge\ninto the main branch is signed by GitHub, so that's something. If this is\ngood enough for Microsoft, is it good enough for us?

\n

Balancing productivity and security​

\n

Commit signing is ultimately about trust: do you trust the committer, and by\nextension, the committed code? This trust is (or should be) backed up with\nquality code reviews, static application security testing (SAST), and other\ndefensive measures, regardless of who created the commit.

\n

Is the productivity benefit of Coding Agent worth the tradeoff on commit\nsigning? We have solid engineering practices, SLSA level 2\ncompliance, and receive high marks from security auditors for the strength of\nour build pipeline. Commit signing is the least of our security measures.

\n

Conclusion​

\n

Coding Agent can be a valuable and productive tool for resolving well documented\nissues quickly, while allowing my team to remain more focused on feature\ndevelopment (with or without Coding Agent). On balance, lifting the enforcement\nof code signing to allow use of Code Agent seems worth the tradeoff.

\n
warning

Please review your organization's security posture carefully before changing\nyour commit signing policy.

\n
\n

Postscript 1: Open-Source problem​

\n

I failed to mention that commit signing is great for regular contributors to an\nopen-source repository, but presents significant friction for new / casual\ncontributors, as described in this feature\nrequest.

\n

Postscript 2: Additional sample repositories​

\n

More examples of prominent open source projects that do not require signed\ncommits on a pull request:

\n", "url": "https://blog.safnet.com/2025/06/17/copilot-commit-signing", "title": "GitHub Copilot and Signed Commits", "summary": "GitHub Copilot [Coding", "date_modified": "2025-06-17T00:00:00.000Z", "tags": [ "programming", "architecture", "security", "technology" ] }, { "id": "https://blog.safnet.com/2025/05/20/initial-experiment-gh-copilot-agent", "content_html": "

GitHub CoPilot recently promoted Agent mode as a full\nfeature\nin VS Code. Many writers and commenters in the blogosphere think competitors are\nstill better; perhaps they're right. But this is the one I have, and Microsoft\nis innovating rapidly - seeing that first hand at Microsoft Build even as I type\nthis. So in recent days I've performed three initial experiments that I want to\nshare. Not because I'm doing anything brilliant: just trying to find tasks that\nmight be a good fit, learn how to interact with the tool, and share a bit to\nhelp others in my proximity.

\n

Lesson: be patient, and be in a learning mode. Not just waiting for it to finish\n(might take several minutes), but also in getting the right results. I remarked\non a failure (below) to my colleague Jason Hoekstra; he suggested that I simply\nundo and try again. It was the right advice. Experiment and build an intuition.

\n

Below I describe three experiments and outcomes.

\n

Converting JSON to YAML​

\n

A giant JSON file: almost 4 MB. \"Self: would this be smaller as YAML? Why not\nask GitHub CoPilot to solve this.\". Prompt, in Agent mode:

\n
Convert this JSON file to YAML
\n

Result: it tried to run a PowerShell command. It failed, saying that\nConvertTo-Yaml doesn't exist. I suspected that I needed to install from the\nPowerShell Gallery. Tried that - manually; I didn't yet have the presence of\nmind to ask CoPilot to fix it for me. I found it in the Gallery, installed,\nre-ran the command, and it worked.

\n

New file size: something like 3.1 MB. Nice.

\n

Assessment: Small goof at first, but easily resolved. It found a simple set of\ncommands to run in PowerShell. I gave no instructions on how to solve this.\nPretty cool that it picked PowerShell on its own.

\n

Replace log4net with Serilog​

\n

Our Admin API application has log4net, and we have talked about converting to\nSerilog. But this bit of tech debt was a low priority for the development team.

\n

This morning...💡see what GitHub CoPilot can do in Agent mode!

\n
Please convert this application from using log4net to using serilog.
\n

Couple of minutes later... success... of a sort. Reviewing the changes, I see\nthat it removed all calls to the logger!

\n

The chat window shows me the steps it took. I see that the first step was to\nremove log4net. It seems to have been overzealous about that removal. Probably\nremoved those lines because the logger was missing, the code couldn't build, and\nit automatically adjusted to fix the code. Oops, undo.

\n

Next, I copied the dotnet add commands that installed Serilog (found in the\nchat window) and then reran them. Then I took a smaller step, with something\nlike this:

\n
Add serilog initialization and configuration to this solution.
\n

This was basically the second step that CoPilot ran before. Note that I didn't\ntell it to remove log4net. But, it was in the same chat history, so CoPilot had\na memory of removing log4net in the past. It went ahead and removed log4net\nafter installing Serilog. Mostly got it right this time.There was one place\nwhere is still removed something that I wanted, so I asked it:

\n
\n

Write a WARNING message to the log when CORS is not enabled.

\n
\n

That worked... but it inefficiently called the ASP.NET builder to create a\nLoggerFactory, then use that factor to create a Logger. Next:

\n
Refactor so that the logger factory is injected into the method signature.
\n

Done. It helpfully asks me if it should update any call sites.

\n
Yes, update the call sites to pass an ILoggerFactory.
\n

Done. After this, I entered additional prompts to help convert\nthe log4net format to a serilog template and to finally remove log4net.

\n

Assessment: Small bites helped. But I might have just tried the original request\nwith slightly more useful instructions, and it may have gotten there on its own.\nMaybe I should have described what the problem was and asked CoPilot to write a\nbetter prompt for me 😁.

\n

Fix a warning​

\n

While testing the log response, I noticed a Warning that occurs on startup. Hey GitHub CoPilot...

\n
My application logs the following warning at runtime. Can you help me solve it?

```
WARNING Microsoft.EntityFrameworkCore.Query [(null)] - Compiling a query which loads related collections for more than one collection navigation, either via 'Include' or through projection, but no 'QuerySplittingBehavior' has been configured. By default, Entity Framework will use 'QuerySplittingBehavior.SingleQuery', which can potentially result in slow query performance. See <https://go.microsoft.com/fwlink/?linkid=2134277> for more information. To identify the query that's triggering this warning call 'ConfigureWarnings(w => w.Throw(RelationalEventId.MultipleCollectionIncludeWarning))
```
\n

A short time later, the problem was fixed. The chat told me about how it setup\n\"query splitting\" behavior in Entity Framework. I wondered to myself, \"why isn't\nthis the default? Is it safe?\" Switching to \"ask\" mode instead of \"agent\":

\n
are there any downsides to switching to this query splitting behavior?
\n

The answer provided me with pros and cons, and a summary of why it feels that\nthis is safe \"for most cases.\" This application does not have heavy load and I\nagree that this looks safe, so I kept the changes.

\n

Assessment: Thanks, CoPilot! Another bit of tech debt quickly solved.

", "url": "https://blog.safnet.com/2025/05/20/initial-experiment-gh-copilot-agent", "title": "Initial Experiments with GitHub Copilot in Agent Mode", "summary": "GitHub CoPilot recently promoted [Agent mode as a full", "date_modified": "2025-05-20T00:00:00.000Z", "tags": [ "programming", "architecture", "dotnet", "technology" ] }, { "id": "https://blog.safnet.com/2025/04/03/docker-as-dev-env", "content_html": "

Containerization of an application benefits operations of the application by\nsolving the problem of \"it works on my machine\" (at least, for the application\nitself). The container holds the operating system and all needed components.\nOnce you have Docker on a host - whether localhost, on-prem data center, or in\nthe Cloud - you can run the application with greater confidence, knowing that\nthe application will execute the same in all environments.

\n

But the benefits of containerization can also shift left in the development\nlifecycle. For example: have you ever needed to revisit an older application,\nand realized that you don't have the SDK on your machine? Instead of installing\nthe SDK locally, you may be able to run the SDK in a Docker\ncontainer.

\n

\"The

Lamium amplexicaule aka\nhenbit, the first flower to appear in my yard this year.

\n

This is article is part 1 in a planned series on using Docker containers in the\nsoftware development lifecycle (SDLC).

\n

Revisiting a .NET Core 3.1 Application​

\n

The Ed-Fi ODS Admin\nApp\n(v2.1.1) is a web application for managing the client credentials needed for\nconnecting to the Ed-Fi\nODS/API for HTTP-based\nexchange of educational data. Version 2.1.1 specifically targeted Ed-Fi ODS/API\n3.x, and it was built with .NET Core 3.1. These products were previously\nsupported by my development teams, and now have been deprecated.

\n

.NET Core natively works in Linux, and Microsoft has provided SDK and runtime\nimages for Docker.We can quickly create a development environment for this\napplication, starting with the correct SDK tag:\nmcr.microsoft.com/dotnet/core/sdk:3.1. Then, we want to support the following\ntasks:

\n
    \n
  1. Build the solution and run unit tests
    2. \n
  2. Run (database) integration tests
    3. \n
  3. Run end-to-end tests
    4. \n
  4. Run the application for exploratory testing
    5. \n
\n\n

Since this article is about shifting left, we will not include containerization\nof the application itself.

\n
tip

Review Microsoft Container Registry to find the\nSDK, runtime, or other .NET images you may need.

\n
warning

Many users, including your author, have had trouble pulling images from\nMicrosoft's registry, receiving error messages like this: ERROR: failed to do request: Head \"https://mcr.microsoft.com/v2/dotnet/aspnet/manifests/8.0\": EOF.\nThis appears to be an unresolved bug in Docker Desktop or in Microsoft's web\nserver configuration, related to IPv6. Turning off\nIPv6 on your network\nconnection resolves the issue.

\n

The Build Environment​

\n

We need a .NET Core 3.1 SDK image, and we need to load the application files\ninto the image. Here is a good starting point, using Ubuntu 20. All commands\nbelow will be shown relative to the Ed-Fi-ODS-AdminApp/eng directory.

\n
docker run -it --rm `
    -v ../Application:/opt/AdminApp/Application:rw `
    mcr.microsoft.com/dotnet/core/sdk:3.1.403-focal@sha256:b51325cd8b3eeb14099ca5db2a40bccc126bbb6fcaeec64fd208a73b8f800eec
\n

This command runs in interactive mode (-it), dropping you into an active Bash\nprompt. On exit, the container will be removed automatically (--rm). The\napplication source files are mapped using read-write mode (:rw)

\n

so that the bin and\nobj directories that will be written by the build process.

\n
warning

When using this statement in PowerShell scripts, replace $pwd with $PSScriptRoot.

\n

Build and Unit Test​

\n

Next, we'll want a script that runs the build and the unit tests. It will also\nbe nice to send the console log output to a file for easier review. Since this\nis a .NET image, it contains PowerShell Core. Writing this in Bash would be\nsimple, but let's stick with PowerShell:

\n
Set-Location /opt/AdminApp/Application

$logFile = \"../logs/build-$(Get-Date -Format \"yyyyMMddHHmm\").log\"

dotnet build --nologo | Tee-Object -FilePath $logFile
dotnet test --nologo --filter FullyQualifiedName~UnitTests | Tee-Object -FilePath $logFile
\n

Save this to a file (run-build.ps1), and map it into the running container.\nAlso, it will be convenient to create volume mounts for the logs and for the\nNuGet packages; otherwise, both will disappear when the container exits. At the\nend of the command we execute the file via pwsh. Finally, remove -it so that\nthe command belows runs the script and exits, instead of simply dropping you\ninto a terminal.

\n
docker run --rm `
    -v $pwd/../Application:/opt/AdminApp/Application:rw `
    -v $pwd/run-build.ps1:/opt/AdminApp/run-build.ps1:ro `
    -v $pwd/packages:/root/.nuget/packages:rw `
    -v $pwd/logs:/opt/AdminApp/logs:rw `
    mcr.microsoft.com/dotnet/core/sdk:3.1.403-focal@sha256:b51325cd8b3eeb14099ca5db2a40bccc126bbb6fcaeec64fd208a73b8f800eec `
    pwsh -File /opt/AdminApp/run-build.ps1
\n
tip

Future consideration:

    \n
  1. The reader may want to explore using a dotnet test\nlogger\nfor alternative output types.
    2. \n
  2. The run-build.ps1 script can also be used directly in a CI environment, so\nthat you are running the exact same command in both the localhost and the\nautomated environments.
    3. \n
\n

Integration Tests​

\n

PostgreSQL Setup​

\n

This application's integration tests require SQL Server 2019 (Docker\nimages) or PostgreSQL\n13, and we will need to install the database\ntables. We will use PostgreSQL.

\n

The integration test container will need to communicate with the database\ncontainer. To this end, we can create a docker network and run both containers\ninside that network.

\n
docker network create adminapp

docker run --rm `
    --name postgres `
    --hostname postgres `
    --network adminapp `
    -e POSTGRES_PASSWORD=mysecretpassword `
    -d `
    postgres:13.20-alpine3.21@sha256:236985828131e95a12914071b944d0e0d21da5281312292747e222845f0ea670
\n

Table Installation​

\n

Next, we need to install the AdminApp database tables. Fortunately, this\napplication was previously containerized. The pre-built database\nimage\nis tagged as edfialliance/ods-api-db-admin:v1.1.0.

\n

In development, we may need to modify the database. The Admin App SQL scripts\nare migration scripts. Any modifications we make will be new migrations, which\ncan run on top of the existing database. Furthermore, the Admin App repository\ncontains PowerShell scripts for installing the SQL migration scripts. With a\nlittle experimentation, I was able to run the database migrations by executing\nthe following script inside the build container.

\n
Set-Location /opt/AdminApp

$config =
    @{
        \"engine\" = \"PostgreSQL\"
        \"databaseServer\" = \"postgres\"
        \"databasePort\" = \"5432\"
        \"databaseUser\" = \"postgres\"
        \"databasePassword\" = \"mysecretpassword\"
        \"useIntegratedSecurity\" = $false
        \"adminDatabaseName\" = \"EdFi_Admin\"
    }

$logFile = \"./logs/dbup-$(Get-Date -Format \"yyyyMMddHHmm\").log\"

./eng/run-dbup-migrations.ps1 $config | Tee-Object $logFile
\n
warning

You may find small problems when running a PowerShell script in Linux that was\nwritten and historically used in Windows: watch out for proper casing on file\nnames, since Linux cares about these things; change \\ to / or use\nJoin-Path to create file paths; and don't use .exe as an extension on\ndownloaded tools. I had to modify one of the existing PowerShell scripts as follows:

\n
$exePath = \"$ToolsPath/$toolName.exe\"
if ([System.Environment]::OSVersion.Platform -eq \"Unix\") {
    $exePath = \"$ToolsPath/$toolName\"
}
\n

Test Execution​

\n

The solution contains two integration test projects:\nEdFi.Ods.AdminApp.Management.Tests and\nEdFi.Ods.AdminApp.Management.Azure.IntegrationTests. We will skip the Azure\ntests for now, focusing only on the \"on-premises\" (or fully containerized)\ndeployment path.

\n

The EdFi.Ods.AdminApp.Management.Tests project contains an appsettings.json:

\n
{
    \"AppSettings\": {
        \"AppStartup\": \"OnPrem\",
        \"ApiStartupType\": \"sandbox\",
        \"XsdFolder\": \"Schema\",
        \"DefaultOdsInstance\": \"DefaultOdsInstance\",
        \"DatabaseEngine\": \"SqlServer\"
    },
    \"ConnectionStrings\": {
        \"Admin\": \"Data Source=.\\\\;Initial Catalog=EdFi_Admin_Test;Integrated Security=True\",
        \"Security\": \"Data Source=.\\\\;Initial Catalog=EdFi_Security_Test;Integrated Security=True\",
        \"OdsEmpty\": \"Data Source=.\\\\;Initial Catalog=EdFi_Ods_Empty_Test;Integrated Security=True\"
    }
}
\n

One way to update these values is to set environment\nvariables,\nwhich will override the appsettings file contents. Like so:

\n
$env:AppSettings__DatabaseEngine=\"PostgreSQL\"
$env:ConnectionStrings__Admin=\"server=postgres;database=EdFi_Admin;username=postgres;password=$pgPass\"
\n

Complete Integration Testing Scripts​

\n

We can call this one run-integration.ps1:

\n
Set-Location /opt/AdminApp

$pgPass=\"mysecretpassword\"

$config =
    @{
        \"engine\" = \"PostgreSQL\"
        \"databaseServer\" = \"postgres\"
        \"databasePort\" = \"5432\"
        \"databaseUser\" = \"postgres\"
        \"databasePassword\" = $pgPass
        \"useIntegratedSecurity\" = $false
        \"adminDatabaseName\" = \"EdFi_Admin\"
    }

$logFile = \"./logs/integration-$(Get-Date -Format \"yyyyMMddHHmm\").log\"

./eng/run-dbup-migrations.ps1 $config | Tee-Object -FilePath $logFile

$env:AppSettings__DatabaseEngine=\"PostgreSQL\"
$env:ConnectionStrings__Admin=\"server=postgres;database=EdFi_Admin;username=postgres;password=$pgPass\"
$env:ConnectionStrings__Security=\"server=postgres;database=EdFi_Security;username=postgres;password=$pgPass\"
$env:ConnectionStrings__OdsEmpty=\"server=postgres;database=EdFi_Ods_Empty_Test;username=postgres;password=$pgPass\"

dotnet test --nologo Application/EdFi.Ods.AdminApp.Management.Tests | Tee-Object -Append -FilePath $logFile
\n

And here is the final script to start this process in Docker:

\n
docker network create adminapp

docker run --rm `
    --name postgres `
    --hostname postgres `
    --network adminapp `
    -e POSTGRES_PASSWORD=mysecretpassword `
    -d `
    edfialliance/ods-api-db-admin:v1.1.0@sha256:258fab94ffbb49bc406b065b074a7154050dbdfaae2626d0570672317c575721

try {
    docker run --rm -it `
        -v $PSScriptRoot/../Application:/opt/AdminApp/Application:rw `
        -v $PSScriptRoot/run-integration.ps1:/opt/AdminApp/run-integration.ps1:ro `
        -v $PSScriptRoot/packages:/root/.nuget/packages:rw `
        -v $PSScriptRoot/logs:/opt/AdminApp/logs:rw `
        -v $PSScriptRoot/../eng:/opt/AdminApp/eng:rw `
        --network adminapp `
        mcr.microsoft.com/dotnet/core/sdk:3.1.403-focal@sha256:b51325cd8b3eeb14099ca5db2a40bccc126bbb6fcaeec64fd208a73b8f800eec `
        pwsh -File /opt/AdminApp/run-integration.ps1
}
finally {
    docker stop postgres
    docker network rm adminapp
}
\n

And here's where the experiment goes ends: the integration test project is\nhard-coded to SQL Server. Furthermore, due to licensing concerns, there is no\nSQL Server base image for edfialliance/ods-api-db-admin. It is possible to\novercome this challenge, but not important: the basic point has been made.

\n

End-to-End Testing​

\n

Sadly, this application did not have any end-to-end tests when this version was\ncreated. If it had them, we might be able to start from the integration\ntesting script, which already sets up the database for testing. We would need to\ninject background startup of the application. Then run the tests. The following\nexample is for manual execution after using docker run --it to start the\ncontainer and activate the command prompt.

\n
Set-Location /opt/AdminApp

$pgPass=\"mysecretpassword\"

$config =
    @{
        \"engine\" = \"PostgreSQL\"
        \"databaseServer\" = \"postgres\"
        \"databasePort\" = \"5432\"
        \"databaseUser\" = \"postgres\"
        \"databasePassword\" = $pgPass
        \"useIntegratedSecurity\" = $false
        \"adminDatabaseName\" = \"EdFi_Admin\"
    }

$logFile = \"./logs/integration-$(Get-Date -Format \"yyyyMMddHHmm\").log\"

./eng/run-dbup-migrations.ps1 $config | Tee-Object -FilePath $logFile

$env:AppSettings__DatabaseEngine=\"PostgreSQL\"
$env:ConnectionStrings__Admin=\"server=postgres;database=EdFi_Admin;username=postgres;password=$pgPass\"
$env:ConnectionStrings__Security=\"server=postgres;database=EdFi_Security;username=postgres;password=$pgPass\"
$env:ConnectionStrings__OdsEmpty=\"server=postgres;database=EdFi_Ods_Empty_Test;username=postgres;password=$pgPass\"

# & at the end runs the application in the background
./Application/EdFi.Ods.AdminApp.Web/bin/Debug/netcoreapp3.1/EdFi.Ods.AdminApp.Web&

# Execute the tests
dotnet test ...

# Call `ps` to find the process ID of the background service
ps

# Now stop that service
kill <pid>
\n

Manual and Exploratory Testing​

\n

Startup a container with port mapping on port 8080, then run the script above up\nthrough EdFi.Ods.AdminApp.Web&.

\n
# Assuming network was previously created
docker network create adminapp

docker run --rm `
    --name postgres `
    --hostname postgres `
    --network adminapp `
    -e POSTGRES_PASSWORD=mysecretpassword `
    -d `
    edfialliance/ods-api-db-admin:v1.1.0@sha256:258fab94ffbb49bc406b065b074a7154050dbdfaae2626d0570672317c575721

docker run -d `
    -v $PSScriptRoot/../Application:/opt/AdminApp/Application:rw `
    -v $PSScriptRoot/run-application.ps1:/opt/AdminApp/run-application.ps1:ro `
    -v $PSScriptRoot/packages:/root/.nuget/packages:rw `
    -v $PSScriptRoot/logs:/opt/AdminApp/logs:rw `
    -v $PSScriptRoot/../eng:/opt/AdminApp/eng:rw `
    --network adminapp `
    -p 8080:8080 `
    mcr.microsoft.com/dotnet/core/sdk:3.1.403-focal@sha256:b51325cd8b3eeb14099ca5db2a40bccc126bbb6fcaeec64fd208a73b8f800eec `
    pwsh -File /opt/AdminApp/run-application.ps1
\n

Open http://localhost:8080 in your browser for manually accessing the web\napplication. Then, don't forget to clean up when you're done:

\n
docker stop postgres
docker network rm adminapp
", "url": "https://blog.safnet.com/2025/04/03/docker-as-dev-env", "title": "Docker Containers in the SDLC: .NET Core SDK", "summary": "Containerization of an application benefits operations of the application by", "date_modified": "2025-04-03T00:00:00.000Z", "tags": [ "programming", "architecture", "dotnet" ] }, { "id": "https://blog.safnet.com/2025/02/16/elephant-in-the-room-ai", "content_html": "

Data engineering gurus Joe Reis and Matthew\nHousley once again led a closing town\nhall at Data Day Texas. Rather than opining from the front, they turned\nthe session over to the wisdom of the crowd. Housley seeded the conversation with a\nsingle question – “what is the elephant in the room?” – and the room was ready\nwith an answer: AI. In particular: what is AI going to do to my job?

\n

Given a room full of strangers, some participants were remarkably open about\ntheir fears. Perhaps knowing that the audience is composed of fellow data geeks\nhelped to establish a sense of vulnerability. These fears were being expressed\nby the people who, in theory, should be the ones developing expertise in using\nAI tooling. But that’s how disruptive the technologies may be: even the data\nexperts are uncertain and afraid.

\n

\"Balloon

Perhaps 20 years from now we'll look back and wonder\nwhat the fuss what all about, just as I wonder what the backstory was on this\nballoon-based scarecrow protecting raspberries in my backyard circa 2006. By\nStephen A. Fuqua.

\n

Or at least some of them are. Some in the audience already see themselves as\nthe ones leading the AI wave in their companies, and others feel more confident\ndue to the (perceived?) productivity boost experienced with their own\napplications of the technology. Many embrace the idea that AI Won’t Replace\nHumans — But Humans With AI Will Replace Humans Without\nAI.

\n

Reading the room, the fear may have been equally about the jobs of the future as\nabout the participants’ own jobs. Some noted that many companies have already\nstopped hiring junior engineers because the LLM assistants are already as good.\nA few weeks after the conference, Salesforce announced no new hiring of\nsoftware engineers in\n2025\ndue to LLM-based productivity boosts. A question on many minds: who will be the\nengineers of the future if we do not develop new engineers today?

\n

Sidenote: it will be interesting to see if the hiring freezes are temporary or\nlasting pauses. Is this perhaps as much a PR ploy as it is an engineering\nmanagement strategy?

\n

Some remarked on the need for humans to provide knowledge engineering:\nstructuring knowledge to provide context and more intelligent capability. Hence\nthe continuing need for data\ncataloging. In discussing education and\ntraining, others commented on the need to focus on semantics – algorithms and\nproblem solving – over syntax. LLM’s can help you write the proper code in\nwhatever language, but you as an engineer need to know how to diagnose a problem\nand describe a solution. Another observation is that these tools may help us\ncollectively fix the massive tech debt found in mission critical, but obsolete,\nsoftware systems.

\n

One individual suggested embracing product management.\nLearn to identify and prioritize features and functionality on your own, weaving\nthem into your AI-assisted tools and products, and then learn how sell and\nsupport those tools for your colleagues’ benefit. But no one dared offer the\nsuggestion of turning themselves into salespeople for AI-assisted software!

\n

Both fear and excitement were palpable. Perhaps the only certainty we can rely\non is that our jobs, and even more so those of the newcomers to the field, will\ncontinue to evolve at an accelerating pace. Which makes me all the more appreciative of\nthe fantastic learning environment that is Data Day Texas.

", "url": "https://blog.safnet.com/2025/02/16/elephant-in-the-room-ai", "title": "Addressing the Elephant in the Room – AI – at the Data Day Texas 2025 Town Hall", "summary": "Data engineering gurus [Joe Reis and Matthew", "date_modified": "2025-02-16T00:00:00.000Z", "tags": [ "technology", "data-and-analytics" ] }, { "id": "https://blog.safnet.com/2025/02/03/data-cataloging-at-ddtx2025", "content_html": "

Notes from three talks on data cataloging at the recent Data Day Texas\n2025

\n\n

The Meta Grid​

\n

Ole Olesen-Bagneux opened the conference with a keynote on the meta\ngrid - a decentralized architecture for data\ncatalogs, inspired by the decentralization of microservices in the API world and\nData Mesh in the world of serving data for analytics and reporting. In short, as\nI understood it: meet the users where they are. The enterprise has many data\nsources; organize a system that supports metadata management at the source,\ninstead of trying to centralize all metadata knowledge.

\n

Empower individuals to serve as reference librarians who know how to navigate\nthe meta grid (which might be low tech), helping others to find the right\nsources for their queries. Stop wasting money on error-prone attempts to gather\nand maintain all metadata in a single location. Documentation of the systems is\nthe key, through diagrams and ADRs. These are the tools of the reference\nlibrarians, both human and LLM-based. \"Talk to\" a trained LLM to find the\nrequired data.

\n

Reactions:

\n\n

Investing in Semantics and Knowledge​

\n

Be curious and empathetic, Juan Sequeda urged us, as\nyou build knowledge systems. Do not lose sight of why (again, people first, tech\nfollows): reach economies of scale with your data by making them reusable,\ncomposable, and extensible. This requires identifying and recording context of\nthe data you are storing (knowledge), and some function that combines data with\nthat knowledge to produce new data and knowledge (semantics).

\n

Back to why: he encouraged applying \"five whys\" type of thinking when approached\nwith a solution, to get to the real problem that needs to be solved. Make\nassumptions explicit. Both the assumptions behind the problem / solution, and\nassumptions about the data themselves. More knowledge: what does null mean in\nthis column? That the value is missing or unknown? That it is constantly\nchanging? That it does not exist? Who knows the answer? Record it in the data\ncatalog.

\n

Sequeda shared about an interesting case study of building a culture around data\nquality, driven by the most basic of incentives: the annual bonus. A company\nidentified data quality problems as costing them lost revenue and extra expense.\nTo turn around the culture, every employee in the company, from the worker who\nmight be eyeballing a measurement instead of being precise to the CEO has to\nimprove the quality of the data, with 25% of their bonus riding on meeting\ncertain benchmarks.

\n

Again with simplicity: do things the \"manual\" way until you can justify a tool.\nSpreadsheets are great. Don't start your journey with an expensive tool. As you\ntrack your data, look for the opportunities for reuse. And, look at who is\nworking with the data. Who uses them? Who affects them? Record these\nobservations in the Data Catalog.

\n

Reactions:

\n\n

We Are All Librarians​

\n

Do you organize (data)? Do you research? Do you educate? Track provenance?\nRetrieve information? Then you are a librarian! Or so says Jessica Talisman.\nThis podcast on building\ntaxonomis\nlooks like it covers similar ground.

\n

Her talk seemed compelling and important... but I admit I got lost at some\npoint. For me, this was too abstract, without sufficient concrete example.\nPerhaps pre-supposing too much? Certainly the keynote speaker enjoyed the talk,\nwith very warm comment and question immediately following.

\n

One idea that did sink in: when cataloging metadata, it helps to control the\nvocabulary. This becomes the taxonomy. Minimize semantic ambiguity by building a\nthesaurus, with a list of terms, associative relationships, and set of rules on\nhow to use it. This seems very powerful for data mapping exercises.

\n

Conclusion​

\n

Conceptually, taxonomy and ontologies should be interesting to me. But I can get\nlost and a bit bored without hands on experience. Nevertheless these sessions\noffered tantalizing clues that one day yet assemble themselves into an aha!\nmoment.

", "url": "https://blog.safnet.com/2025/02/03/data-cataloging-at-ddtx2025", "title": "Data Cataloging at Data Day Texas 2025", "summary": "Notes from three talks on data cataloging at the recent [Data Day Texas", "date_modified": "2025-02-03T00:00:00.000Z", "tags": [ "technology", "data-and-analytics" ] }, { "id": "https://blog.safnet.com/2025/01/24/openapi-to-csv", "content_html": "

Following up on Grudgingly Accepting AI Coding Assistants, I have a small win from my\nfirst experiment with GitHub Copilot: a Python script for converting portions of\nan Open API specification file to a CSV. There is nothing revolutionary in this\nsmall win; countless programmers have doubtlessly done much more. But a\nreluctant engineer has to start somewhere.

\n

\"Screenshot

\n
tip

Postscript 3/22/2025: Now available in an expanded form, supporting both Open API version 2 (Swagger)\nand version 3, at openapi-to-csv

\n
warning

6/25/2025: On further testing, I discovered significant problems with this\napproach, which had looked so promising. I should try again some day with a\nbetter prompt - I don't blame Copilot, I blame my skill at using it.

\n

Context​

\n

Interoperability with the Ed-Fi Data\nStandard is mediated through a\nREST API, which is documented via an Open API\nspecification. Conversion of portions of the specification to CSV could simplify\ncomparison to other specifications and/or ingestion into a data catalog.

\n

The specification file describes the HTTP-based interactions available in the\nAPI application, including the schemas for the POST and PUT request bodies\nand GET response bodies. These schemas are, in practice, the tangible\nrealization of the Ed-Fi Data Standard. Thus, a CSV file containing the a\nlisting of all of the properties for each path is a listing of the elements of\nthe effective Ed-Fi Data Standard.

\n
info

There are small divergences between the Data Standard and this API\nSpecification, such as how school years are represented. However, since actual\nexchange of data occurs through the API, then the API specification is the\n\"physical\" source of truth that we need to work with.

\n

Example​

\n

Below is a greatly truncated extract from the Ed-Fi API spec:

\n
{
  \"openapi\": \"3.0.1\",
  \"info\": {},
  \"servers\": [],
  \"paths\": {
    \"/ed-fi/absenceEventCategoryDescriptors\": {
      \"get\": {
        \"summary\": \"Retrieves specific resources using the resource's property values (using the \\\"Get\\\" pattern).\",
        \"description\": \"This GET operation provides access to resources using the \\\"Get\\\" search pattern.  The values of any properties of the resource that are specified will be used to return all matching results (if it exists).\",
        \"responses\": {
          \"200\": {
            \"description\": \"The requested resource was successfully retrieved.\",
            \"content\": {
              \"application/json\": {
                \"schema\": {
                  \"type\": \"array\",
                  \"items\": {
                    \"$ref\": \"#/components/schemas/edFi_absenceEventCategoryDescriptor\"
                  }
                }
              }
            }
          },
        }
      }
    }
  }
}
\n

The $ref line points a reusable definition of the object used in GET\nresponses and POST and PUT requests (again, truncated for brevity):

\n
 \"components\": {
    \"schemas\": {
      \"edFi_absenceEventCategoryDescriptor\": {
        \"type\": \"object\",
        \"properties\": {
          \"id\": {
            \"type\": \"string\",
            \"description\": \"\"
          },
          \"codeValue\": {
            \"maxLength\": 50,
            \"type\": \"string\",
            \"description\": \"A code or abbreviation that is used to refer to the descriptor.\",
            \"x-Ed-Fi-isIdentity\": true
          },
          \"namespace\": {
            \"maxLength\": 255,
            \"type\": \"string\",
            \"description\": \"A globally unique namespace that identifies this descriptor set. Author is strongly encouraged to use the Universal Resource Identifier (http, ftp, file, etc.) for the source of the descriptor definition. Best practice is for this source to be the descriptor file itself, so that it can be machine-readable and be fetched in real-time, if necessary.\",
            \"x-Ed-Fi-isIdentity\": true
          },
          \"description\": {
            \"maxLength\": 1024,
            \"type\": \"string\",
            \"description\": \"The description of the descriptor.\",
            \"x-nullable\": true
          }
        }
      }
\n

Putting this information together, the desired output would be:

\n
PathProperty NameProperty TypeDescription
/ed-fi/absenceEventCategoryDescriptorsidstring
/ed-fi/absenceEventCategoryDescriptorscodeValuestringA code or abbreviation...
/ed-fi/absenceEventCategoryDescriptorsnamespacestringA globally unique namespace...
/ed-fi/absenceEventCategoryDescriptorsdescriptionstringThe description of the...
\n

AI Assistant's Output​

\n

I've not had time to do much work in Python for the past year, so I'm a bit\nrusty. And, I'm very pressed for time. 💡Light bulb moment: maybe an AI\nassistant can help me out. Let's give GitHub Copilot a shot at this.

\n
\n

Using the attached Open API input file as an example, generate a Python\nprogram that converts this file into a CSV file with one row for every schema\nproperty, and columns for the schema name, property name, property type, and\ndescription. Also add a column for the path associated with that schema\ncomponent.

\n
\n

The prompt was easy to write and feels natural. I don't really need the schema\nname, but I suspect it will be useful. The last part, about appending the Path,\npurposefully assumes an understanding of OpenAPI: namely, how to connect the\ndots from a schema back up to path, via $ref. I do not expect success with\nthe Path.

\n

The code output by GitHub Copilot is listed in the appendix\nbelow. I run it. A short time later, it finishes with a clean\nexit and a shiny new CSV file in my directory.

\n

But is it right?

\n
schema_nameproperty_nameproperty_typedescriptionpath
edFi_absenceEventCategoryDescriptoridstring/ed-fi/absenceEventCategoryDescriptors
edFi_absenceEventCategoryDescriptorcodeValuestringA code or.../ed-fi/absenceEventCategoryDescriptors
edFi_absenceEventCategoryDescriptornamespacestring\"A globally.../ed-fi/absenceEventCategoryDescriptors
edFi_absenceEventCategoryDescriptordescriptionstringThe description.../ed-fi/absenceEventCategoryDescriptors
edFi_academicHonorCategoryDescriptoridstring/ed-fi/absenceEventCategoryDescriptors
edFi_academicHonorCategoryDescriptorcodeValuestringA code or.../ed-fi/absenceEventCategoryDescriptors
edFi_academicHonorCategoryDescriptornamespacestring\"A globally.../ed-fi/absenceEventCategoryDescriptors
edFi_academicHonorCategoryDescriptordescriptionstringThe description.../ed-fi/absenceEventCategoryDescriptors
edFi_academicSubjectDescriptoridstring/ed-fi/absenceEventCategoryDescriptors
edFi_academicSubjectDescriptorcodeValuestringA code or.../ed-fi/absenceEventCategoryDescriptors
edFi_academicSubjectDescriptornamespacestring\"A globally.../ed-fi/absenceEventCategoryDescriptors
edFi_academicSubjectDescriptordescriptionstringThe description.../ed-fi/absenceEventCategoryDescriptors
\n

It is! (Sidenote: the quotation marks on the description of namespace are due to the\npresence of commas). How did it get the paths right? Let's look at a bit of code:

\n
    for path, path_details in paths.items():
        for schema_name, schema in schemas.items():
            rows.extend(process_schema(schema_name, schema, path))
\n

It simply iterates over the path elements 🤔. Let's look at the output again.\nOh.... look carefully at the last entry shown above: it has the wrong path. My\ninitial enthusiasm short-circuited my quality assurance brain and I stopped\nlooking carefully after the first several rows. The correctness of those first\nrows was just happenstance, from the alphabetic order of the elements in the\nspecification file.

\n

Revision​

\n

Can I revise the prompt in a way that will instruct Copilot on how to navigate\nOpenAPI?

\n

Perhaps. But why bother? Time for the human to take over. We can easily convert\nthe schema_name column into the desired content from the path column.\nTelling Copilot how to do this would take me longer than simply writing it in\nPython myself.

\n
(namespace, entity) = schema_name.split(\"_\", 1)
path = f\"/{namespace}/{entity}\"
\n

Conclusion​

\n

Given my rust, Copilot saved me 15 to 30 minutes of jogging my memory and\niterating over the small mistakes I likely would have made. It took me most of\nthe way, and I was able to take over quickly to finish the job.

\n

A more skillful engineer could surely instruct the assistant on how to navigate the\nOpen API specification to achieve the desired goal. But this is entirely\nunnecessary; it would have been a poor use of time and electricity.

\n

It was a simple script that will help avoid hours of tedious, manual labor to\nconvert the JSON to CSV. The stakes were low - this is not a production\napplication, the script size is small, and reinforcing my Python skills is not a\npresent priority - and yet this felt like a small win that even this reluctant\nuser is happy to accept.

\n

Appendix: the Code​

\n

Original Output​

\n
import json
import csv

def extract_schemas(openapi_data):
    schemas = openapi_data.get('components', {}).get('schemas', {})
    return schemas

def extract_paths(openapi_data):
    paths = openapi_data.get('paths', {})
    return paths

def process_schema(schema_name, schema, path):
    rows = []
    properties = schema.get('properties', {})
    for prop_name, prop_details in properties.items():
        row = {
            'schema_name': schema_name,
            'property_name': prop_name,
            'property_type': prop_details.get('type', 'N/A'),
            'description': prop_details.get('description', 'N/A'),
            'path': path
        }
        rows.append(row)
    return rows

def main():
    with open('swagger.json', 'r') as f:
        openapi_data = json.load(f)

    schemas = extract_schemas(openapi_data)
    paths = extract_paths(openapi_data)

    rows = []
    for path, path_details in paths.items():
        for schema_name, schema in schemas.items():
            rows.extend(process_schema(schema_name, schema, path))

    with open('output.csv', 'w', newline='') as csvfile:
        fieldnames = ['schema_name', 'property_name', 'property_type', 'description', 'path']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

        writer.writeheader()
        for row in rows:
            writer.writerow(row)

if __name__ == '__main__':
    main()
\n

Modified Version​

\n

With additional modification to remove schema_name and place path as the\nfirst column.

\n
import json
import csv

def extract_schemas(openapi_data):
    schemas = openapi_data.get('components', {}).get('schemas', {})
    return schemas

def convert_to_path(schema_name):
    (namespace, entity) = schema_name.split(\"_\", 1)
    return f\"/{namespace}/{entity}\"

def process_schema(schema_name, schema):
    rows = []
    properties = schema.get('properties', {})
    for prop_name, prop_details in properties.items():
        # Only interested in schemas of form \"<namespace>_<entity>\". Lazily just check for underscore.
        if not \"_\" in schema_name:
            continue

        row = {
            'path': convert_to_path(schema_name),
            'property_name': prop_name,
            'property_type': prop_details.get('type', 'N/A'),
            'description': prop_details.get('description', 'N/A')
        }
        rows.append(row)
    return rows

def main():
    with open('swagger.json', 'r') as f:
        openapi_data = json.load(f)

    schemas = extract_schemas(openapi_data)

    rows = []
    for schema_name, schema in schemas.items():
        rows.extend(process_schema(schema_name, schema))

    with open('output.csv', 'w', newline='') as csvfile:
        fieldnames = ['path', 'property_name', 'property_type', 'description']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)

        writer.writeheader()
        for row in rows:
            writer.writerow(row)

if __name__ == '__main__':
    main()
", "url": "https://blog.safnet.com/2025/01/24/openapi-to-csv", "title": "OpenAPI to CSV with Help from GitHub Copilot", "summary": "Following up on Grudgingly Accepting AI Coding Assistants, I have a small win from my", "date_modified": "2025-01-24T00:00:00.000Z", "tags": [ "programming", "technology", "ed-fi" ] }, { "id": "https://blog.safnet.com/2025/01/14/accepting-ai-assistants", "content_html": "

As a software engineering director building open source products, I have\nprohibited my teams from using AI coding assistants due to concerns about\nintellectual property and questions about the risks and real world effectiveness\nof AI coding assistants. It is now time to allow and even encourage AI coding\nassistants, with guardrails.

\n

\"Balcones

Balcones National Wildlife Refuge, December 2024, by Stephen A. Fuqua

\n

Intellectual Property​

\n

My fundamental argument against LLM-based coding assistants was this:

\n
    \n
  1. These LLMs are trained on open source software.
    2. \n
  2. That training and/or the use of code created from that trained model may be a\nviolation of their license agreements.
    3. \n
  3. My teams build open source software and we do not want our license violated.
    4. \n
  4. Golden rule: do unto others as you would have them do unto you.
    5. \n
  5. Ergo, do not use coding assistants.
    6. \n
\n

Not surprisingly, many others feel the same, as noted in The\nRegister\n(2023-05-12):

\n
\n

Software developers have bristled that Codex and Copilot were created from\ntheir code, and sometimes reproduce it, without explicit permission or concern\nfor the terms under which they licensed their work. And some of them have sued\nover it.

\n
\n

But the developers suing over GitHub Copilot got dealt a major blow in\ncourt\n(The Verge, 2024-07-09) — although the open source license violation is\nstill an open question.

\n

At least two coding assistants, Tabnine and Amazon\nQ Developer (fka Code Whisperer), offer up\nattributions that you can paste into your open source notifications. GitHub has\nalso added public code filtering and now reference\nlinks\nwhen public code filtering is off, which may satisfy the attribution /\nnotification problem. Whether the training is fully legal given the open\nsource licenses remains as an open risk. However, I do not seriously believe that the\ncourt system will strike down LLMs on the basis of being trained on others'\nintellectual property.

\n

With multiple free and low-cost tools available, and no way to police individual\nusage, it has long been a moot point. People are probably using these tools\nalready, unbeknownst to me, even though I've asked them not to. Time for me to\nstop fighting this and focus on safe and effective usage.

\n

Effectiveness​

\n

Are these tools effective? There's the marketing, and then there's real world\nexperience. The articles I've been reading suggest both sides of things: that\nthe code produced by AI assistants can be garbage, and that it can be helpful.\nThe code likely will require judicious editing from a trained and talented\nengineer. And, that engineer will need to be talented not only in the faculty of\nunderstanding the code, but also in learning how to provide meaningful\ninstructions\nto the AI assistant.

\n

As many other writers have noted, a world of AI coding assistants could lead to a\nworld where junior developers never learn to code well and thus have trouble\nimproving the code produced by the assistant. Or have trouble recognizing major\nsecurity flaws in the code (Predictions 2024: Security And Risk Pros Will Apply\nGuardrails Beyond Regulatory\nMandates\nfrom Forrester). Even seasoned veterans can easily overlook security problems\nif they review too quickly.

\n

Last fall, I installed Amazon Q Developer and asked it to help me write\nadditional unit tests for a C# class that had been neglected. It helped me\nuncover several important edge cases around null handling. I failed to tell it that\nI wanted my tests in NUnit and that I prefer use of FluentAssertions; it gave me\nXUnit that I then converted to NUnit (admittedly I cannot remember if a\nlimitation prevented me from tuning my request, or if I was just annoyed and\nwanted to limit my interactions).

\n

Overall, I estimate it saved me about 20 minutes time on that task. And it\nhelped find a low probability bug in the system-under-test that I may have\nmissed. That was useful.

\n

Guardrails​

\n

We will continue to enforce existing practices:

\n
    \n
  1. Use of linters (JavaScript, Python, PowerShell) and\nRoslyn-based\ncode analyzers for C# (tip:\nSonarlint and\nsonar-dotnet).
    2. \n
  2. Use of static application security tools (e.g. CodeQL) on all contributions\nto our open source code, whether made by staff, contractors, or the public.
    3. \n
  3. Mandatory peer review and merge process into the protected main branch that\ngenerates our binary releases.
    4. \n
\n

To complement these, we must encourage and support developers to strengthen\ntheir own habits as well as their abilities to sniff out and question\npotentially harmful or simply dodgy code. And we should encourage a community of\npractice where developers on different teams freely share about their\nexperiences.

\n
danger

Those who are working with closed source applications will want to read the fine\nprint to understand the privacy and security implications of your use of an AI\ncoding assistant. For example, GitHub\nsays: \"GitHub does not use either Copilot\nBusiness or Enterprise data to train its models.\". This seems to imply that it\ndoes use data from Free and Pro.

\n
tip

Also see GitHub Copilot Security and\nPrivacy Concerns: Understanding the Risks and Best\nPractices\nfor more detailed security tips.

\n

Personal Plans​

\n

Two weeks into the new year, I have yet to write a line of code. Unless you\nconsider Markdown files and Mermaid diagrams \"documentation as code\". At some\npoint, once the new year kickoff tasks are complete, I hope to jump into the\nfray and complete a few programming tasks. When I do, I plan to continue\nevaluating Amazon Q and may try out GitHub Copilot and Tabnine out of\ncuriosity. (Only tools that support VS Code make it on my evaluation list).

\n

Depending on the task at hand, I would like to experiment with these uses:

\n
    \n
  1. Generating unit tests. I practice and encourage Test Driven\nDevelopment. But not everyone does.\nLow-hanging fruit for me to contribute, without distracting the team: help\nthem beef up the code coverage. (Moral Hazard Warning: only cover older code\nor expand existing tests. Don't let this be an excuse for not testing!).
    2. \n
  2. Asking the assistant to review my code. Amazon\nQ,\nGitHub\nCopilot,\nTabnine.
    3. \n
  3. Generating boilerplate code. Although this is a popular usage, we have\nlittle boilerplate code left to write. Our applications are at the stage of\nneeding real thought and design in order to grow into the next user story.
    4. \n
\n

Parting Words​

\n

To all of the developers I work with: go forth and write quality\ncode, with or without an AI assistant.

", "url": "https://blog.safnet.com/2025/01/14/accepting-ai-assistants", "title": "Grudgingly Accepting AI Coding Assistants", "summary": "As a software engineering director building open source products, I have", "date_modified": "2025-01-14T00:00:00.000Z", "tags": [ "programming", "technology" ] }, { "id": "https://blog.safnet.com/2024/12/31/recommended-reading-new-managers", "content_html": "

Manager-dom was not a sought-after status (though I admit to prior curiosity),\nbut as with many other engineers-turned-managers, it found me anyway (Q3 2023).\nBeing a team lead and then an architect came naturally and brought success for\nmany years. Some of the skills translate well from those roles while other\nnecessary capabilities tend to be underdeveloped. Nothing can truly substitute\nfor the value of gaining experience and reflecting on it; here are a few\nresources that I have found useful in my journey so far, giving me new ideas to\nplay with and questions to aid my reflection.

\n

\"Sharp-shinned

Sharp-shinned Hawk at Resaca de la Palma State Park, by Stephen A. Fuqua (2024).

\n

Let's start with What I wish somebody told me during my first year as a\nmanager,\nwhich I only encountered yesterday. Most of its advice is not new, and yet\nit is nevertheless timely. What I most appreciated was this acknowledgment:

\n
\n

\"You didn’t expect the emotional drain, right? That some days you’ll be so\nexhausted from being responsible for everything, that you’ll come home and\njust collapse on the couch.\" (Anton Zaides)

\n
\n

For me it wasn't so much a collapse on the couch as a recognition that it became\nharder to get a second (third?) wind after dinner and on the weekend, at least\nfor creative endeavors or mentally challenging avocational work.

\n

Here are two books that offer mutually-reinforcing advice, though in wildly\ndifferent writing styles:

\n\n

Both books have topics addressing the root causes of the emotional drain.\nSometimes you can't avoid the drain, which probably means having a difficult\nconversation. Crucial Conversations: Tools for Talking When Stakes Are\nHigh\nwill help you move past the fight-or-flight tendency into a productive\ninteraction.

\n

Finally, at some point I skimmed through Management 3.0: Leading Agile\nDevelopers, Developing Agile\nLeaders, a book I had first read\nten years ago (cf in this blog: easy : simple :: lazy :\nefficient and Management\n3.0: Knowledge and\nDiversity). A\nuseful resource, but, unfortunately, what currently stands out in this reader's\nmind is the author's off-putting sarcasm.

", "url": "https://blog.safnet.com/2024/12/31/recommended-reading-new-managers", "title": "Tools for Engineering New Managers", "summary": "Manager-dom was not a sought-after status (though I admit to prior curiosity),", "date_modified": "2024-12-31T00:00:00.000Z", "tags": [ "books", "meta" ] }, { "id": "https://blog.safnet.com/2024/06/24/living-with-agile", "content_html": "

In reaction to all of the \"agile is dead\" articles, I am cleaning up old posts\nabout Agile, re-reading them, contemplating lessons learned but\nforgotten, and asking myself if some practices have outlived their usefulness.\nThat is the spirit of agility: the interplay of action and reflection.

\n

\"Agile\" is not a silver bullet\nfor improving software productivity, reliability, and simplicity. But \"Agile\"\ncontinues to give us tools that can foster improved software engineering.

\n

The Agile Manifesto was a distillation of certain trends that the authors had\nnoticed in their successful projects. On balance, they ring true to my\ntwenty-five year career in software.

\n

It did not offer guarantees and it did not offer to solve world hunger. Through\nprescriptive frameworks such as Extreme Programming and Scrum, and common\npractices such as development of story cards and short cycles (aka sprints), the\nAgile \"revolution\" broke us free from the confines of gigantic requirements and\ndesign documents that were always at least slightly wrong, and frequently very\ndifficult to change. It helped us embrace the uncertainty of software\ndevelopment, empowering us to find our way out of that wrongness more quickly\nand productively.

", "url": "https://blog.safnet.com/2024/06/24/living-with-agile", "title": "Living with Agile", "summary": "In reaction to all of the \"agile is dead\" articles, I am cleaning up [old posts", "date_modified": "2024-06-24T00:00:00.000Z", "tags": [ "technology", "programming", "agile" ] }, { "id": "https://blog.safnet.com/2024/05/01/bahai-principles-for-climate-action", "content_html": "

Amongst its many admirable principles, where does one find environmental\nconcerns in the Bahá'í Faith? So I wondered, early on my journey on this\nreligious path. Recently, in honor of Faith Climate Action\nWeek, several\nfriends and I reflected on this question.

\n

Our conversation opened with a brief meditation on this passage of praise for\nGod, who is recognized through all the natural wonders:

\n
\n

“… whatever I behold I readily discover that it maketh Thee known\nunto me, and it remindeth me of Thy signs, and of Thy tokens, and of Thy\ntestimonies. By Thy glory! Every time I lift up mine eyes unto Thy heaven, I\ncall to mind Thy highness and Thy loftiness, and Thine incomparable glory and\ngreatness; and every time I turn my gaze to Thine earth, I am made to\nrecognize the evidences of Thy power and the tokens of Thy bounty. And when I\nbehold the sea, I find that it speaketh to me of Thy majesty, and of the\npotency of Thy might, and of Thy sovereignty and Thy grandeur. And at whatever\ntime I contemplate the mountains, I am led to discover the ensigns of Thy\nvictory and the standards of Thine omnipotence.”

\n

Bahá'u'lláh, Prayers and Meditations by Bahá’u’lláh, CLXXVI

\n
\n

\"Naw

Wildflowers on Naw Ruz at Enchanted Rock State Natural Area, by Stephen A. Fuqua (2024).

\n

From there, we turned to the importance of thinking about the oneness of\nhumanity, and the analogy: if one limb of the body is afflicted, the entire\nhuman body suffers. So too is it with individuals in a community. Thus, knowing\nthat the climate crisis afflicts humanity, we must act to avert that crisis as\nbest we can.

\n

Bahá'í literature and teachings first emphasize the principals of the \"three\nonenesses\" (or \"unities\"): that of\nhumanity\nis joined by recognition of the oneness of\nGod\nand of\nreligion.\nNext, they frequently (and rightly) echo\nprinciples\nenunciated by 'Abdu'l-Bahá in His writings and talks, dating especially to the\nfirst decades of the twentieth century. Lists of those principles frequently\nreference abolition of prejudice, equality of the sexes, independent\ninvestigation of the truth, harmony of science and religion, world peace,\ninternational governance, universal auxiliary language, and compulsory\neducation. Ther are no forewarnings about health impact of pollution, the future\nclimate calamity, or loss of biodiversity.

\n

On the one hand: these were not the most pressing challenges of the time. And on\nthe other: these very principles, and the institutions inaugurated by\nBahá'u'lláh and 'Abdu'l-Bahá, are essential for solving the new questions and\nchallenges that have arisen since that time.

\n

Not surprisingly, we quickly identified \"harmony of science and religion\" as a\nkey concept in thinking about climate change and related issues . This principle\nsuggests that religion bereft of science – rationalism, investigation\n– tends toward superstition and social stagnation. And science and\ntechnological innovation without spirituality and religion tends toward a\nmaterialism that demeans individual dignity, facilitates an uneven distribution\nof wealth, and encourages excessive consumption.

\n

Clearly there is a very strong link between excess consumption, pollution, and\nclimate degradation. Though not in that list above, Bahá'u'lláh mentioned the\nimportance of \"moderation\" many times. A favorite and germane example:

\n
\n

“The civilization, so often vaunted by the learned exponents of arts and\nsciences, will, if allowed to overleap the bounds of moderation, bring great\nevil upon men. Thus warneth you He Who is the All-Knowing. If carried to\nexcess, civilization will prove as prolific a source of evil as it had been of\ngoodness when kept within the restraints of moderation.”

\n

Bahá'u'lláh, Gleanings from the Writings of Bahá'u'lláh, CLXIV

\n
\n

We noted that fundamental concepts of personal spirituality naturally moderate\nour desires for physical gratification and consumption. The book Reflections on\nthe Life of the Spirit is the first unit in a sequence of\ncourses for youth and\nadults. The book is studied by Bahá'ís and their friends worldwide. Under unit\nheadings of \"Understanding the Bahá'í Writings\", \"Prayer\", and \"Life and Death\",\nthis book introduces spiritual practices of reading sacred scriptures daily;\nholding ourselves accountable for our thoughts and deeds; personal and communal\n(not congregational) prayer; and the notion of developing the life of the soul.\nSurely these are ameliorating influences to counter \"civilization\" \"carried to\nexcess.\"

\n

\"A

Central Texas Hill Country sunset, by Stephen A. Fuqua (2024).

\n

Another fundamental concept in the Faith, not listed in the principles above, is\nthe idea of\nconsultation:

\n
\n

“The Great Being saith: The heaven of divine wisdom is illumined with\nthe two luminaries of consultation and compassion. Take ye counsel together in\nall matters, inasmuch as consultation is the lamp of guidance which leadeth\nthe way, and is the bestower of understanding.”

\n

Bahá'u'lláh, Tablets of Bahá'u'lláh, Lawḥ-i-Maqṣúd

\n
\n

This spiritual consultation is the idea of a group of people engaged in a frank\nand cordial search for the truth. It is a cornerstone of family, community, and\ninstitutional decision-making in the Bahá'í Faith, and a tool that Bahá'ís offer\ngladly to any who wish to make use of it.

\n

As we wound down our conversation, we noted how the knowledge, attribute, and\nactions imparted by all of the books in the sequence of courses teach lessons\nvaluable to those who recognize the need for improved environmental stewardship.\nFor example, the topic of consultation is treated many times, most especially in\nthe tenth book. The second book introduces the idea of \"meaningful\nconversations\" – aiding the participant to develop the skill of having\nthis very sort of conversation. Likewise each book could be linked in some way.

\n

Finally, we touched briefly on the importance of action: recognizing the\nimperative for the course of humanity to be shifted onto a sustainable path, we\nmust each determine how we can contribute to individual action while\nsimultaneously supporting collective endeavors.

\n

Other Resources​

\n", "url": "https://blog.safnet.com/2024/05/01/bahai-principles-for-climate-action", "title": "A Conversation on Bahá'í Principles for Climate Action", "summary": "Amongst its many admirable principles, where does one find environmental", "date_modified": "2024-05-01T00:00:00.000Z", "tags": [ "religion", "nature", "bahai-faith", "discourse", "climate" ] }, { "id": "https://blog.safnet.com/2024/04/18/cloud-native-edi-fi-api", "content_html": "
\n

\"For the past two years, the Ed-Fi Alliance software development team has been\nlistening to community members through its Technical Advisory Group, Special\nInterest Groups, and at our annual events. We have been hearing that the pace\nof change in the Ed-Fi ODS/API Platform needs to accelerate, shifting to a\ncloud-native architecture that can better support large-scale deployments\nwhile offering greater cost and performance flexibility. To do so, we need a\nreboot.\"

\n
\n

→ Full article at New Cloud-Native Functionality Coming to the Ed-Fi Alliance\nTechnology Suite

\n

Though barely mentioned in the article, the work to create a production ready\nsystem has been dubbed Project\nTanager, the third\nbird-related project name in my tenure with the Ed-Fi Alliance\n(Roadrunner,\nMeadowlark).

\n

\"Scarlet

", "url": "https://blog.safnet.com/2024/04/18/cloud-native-edi-fi-api", "title": "Project Tanager, the next generation of Ed-Fi API software", "summary": "\"For the past two years, the Ed-Fi Alliance software development team has been", "date_modified": "2024-04-18T00:00:00.000Z", "tags": [ "programming", "ed-fi", "architecture" ] }, { "id": "https://blog.safnet.com/2024/02/11/delivering-value-as-engineer", "content_html": "

In Unpopular Opinion: It’s harder than ever to be a good software\nengineer,\nauthor Juraj Malencia opines that \"good engineering\" equates to \"bring[ing]\nmaximum value in achieving a goal.\" This is a comforting thought for those of us\nperiodically wondering if we're still good software or data engineers, though we\nmight not be up-to-date on the latest hype.

\n

Pointing out that \"engineering does not equal programming\", implying that we can\nbring great \"engineering\" (solutioning) value without writing a line of code, he\npresents a modified version of a venn\ndiagram\non how people in various roles spend their time between programming, alignment,\npeople, and \"other\". I was bemused to note I currently sit squarely in the\ncategory labeled \"beware\" 🤨. A position that may continue throughout this year,\nbut will need to change over time.

\n

One way that we can deliver value as senior engineers is by knowing when, and\nhow, to break the rules. Although I am now in management, I am staying close to\nthe work of one of our development teams. When I saw them struggling to meet the\nsprint commitment, I decided to lean in. There was a tangled situation with\nlibrary dependencies. The thorniest problem lay with library A, which depended\non a dozen or so classes in library B. Remembering a lesson I learned from a\nvery smart \"junior\" developer a few years ago, I decided that Write Everything\nTwice (WET) trumped Don't Repeat Yourself (DRY) in this case.

\n

In other words, I just copied the required code from library B directly into\nlibrary A. In this case, all of the copied code was highly stable, and generally\nof a \"utility\" nature rather than being business logic. Problem solved.

\n

However: I probably could have contributed similar value by simply suggesting\nthat the team duplicate that code, rather than taking it on myself. It felt good\nto do the work, and helped unblock the team, but it might not have been the\nmaximum value I could have brought to the business. What was I not getting\ndone - the opportunity cost? This is a critical lesson for this first-time\nmanager to learn: think twice (or more) before jumping into the code. Get better\nat empowering than (hands-on) solving.

\n

\"Alexander

Maturino da Firenze. Alexander Cutting the Gordian\nKnot,\n1510-1527. The Art Institute of Chicago.

", "url": "https://blog.safnet.com/2024/02/11/delivering-value-as-engineer", "title": "Delivering Value as a Software Engineering Manager", "summary": "In [Unpopular Opinion: It’s harder than ever to be a good software", "date_modified": "2024-02-11T00:00:00.000Z", "tags": [ "programming" ] }, { "id": "https://blog.safnet.com/2023/12/08/edfi-client-from-swagger", "content_html": "

Motivation​

\n

The Ed-Fi ODS/API is a REST\nAPI that support interoperability of student data systems. The API definition,\nvia the Ed-Fi Data\nStandard, is\nextensible: many large-scale or specialized implementations add their own local\nuse cases that are not supported out of the box by the Ed-Fi Data Standard\n(Extensions). Furthermore, the Data Standard receives regular updates; sometimes\nthese are merely additive, and from time to time there are breaking changes.\nThese factors make it impossible to create a one-size fits all client\nlibrary.

\n

But, not all is lost: the ODS/API exposes its API definition using\nOpenAPI, and we can use Swagger\nCodegen to build a client library\nbased on the target installation's data model / API spec. The basic process of\ncreating a C# code library (SDK) is described in Ed-Fi documentation at Using\nCode Generation to Create an\nSDK\n(Note: this link is for ODS/API 7.1, but the instructions are essentially the\nsame for all versions).

\n

But what about Python? Yes, Swagger Codegen supports Python output. But it is\nnot quite enough - you also need to manage authentication on your own. And,\nrunning Swagger Codgen requires the Java Development Kit (JDK). The notes below\nwill walk through generating a client library with help from Docker (no local\ninstall of the JDK required) and demonstrate basic usage of a simple\nTokenManager class for handling authentication.

\n
tip

See\nEd-Fi-API-Client-Python\nfor a source code repository containing the TokenManager class listed below,\nwhich might receive updates after this post has been published.

\n

Generating an Ed-Fi Client Package​

\n

The Swagger Codegen tool is available as a pre-built Docker\nimage,\nat repository swaggerapi/swagger-codegen-cli. We will use it to build a\nclient package for working with Ed-Fi Data Standard v5.0, which is available through\nEd-Fi ODS/API v7.1. The ODS/API Landing Page has links\nto the Swagger UI-based \"documentation\" (UI on top of OpenAPI specification) for\nall currently supported versions of the ODS/API. From there, we can find a link\nto the specification\ndocument.

\n

The example shell commands use PowerShell, and they are easily adaptable to Bash\nor another shell. The generated code will be in a new edfi-client directory.\nNote that this repository's .gitignore file excludes this directory from\nsource control, since the original intent of this repository is to provide\ninstructions, not a full-blown client. If you fork this repository and want to\ncreate your own package, then you may wish to remove that line from .gitignore\nso that you can keep your custom client code in your forked repository.

\n
$url = \"https://api.ed-fi.org/v7.1/api/metadata/data/v3/resources/swagger.json\"
$outputDir = \"./edfi-client\"
New-Item -Path $outputDir -Type Directory -Force | out-null
$outputDir = (Resolve-Path $outputDir)
docker run --rm -v \"$($outputDir):/local\" swaggerapi/swagger-codegen-cli generate `
    -i $url -l python -o /local
\n

On my machine, this took about a minute to run. Here's what we get as output:

\n
> ls edfi-client

    Directory: C:\\source\\Ed-Fi-API-Client-Python\\edfi-client


Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
d-----        11/27/2023   9:31 PM                .swagger-codegen
d-----        11/27/2023   9:31 PM                docs
d-----        11/27/2023   9:32 PM                out
d-----        11/27/2023   9:31 PM                swagger_client
d-----        11/27/2023   9:31 PM                test
-a----        11/27/2023   9:31 PM            786 .gitignore
-a----        11/27/2023   9:31 PM           1030 .swagger-codegen-ignore
-a----        11/27/2023   9:31 PM            359 .travis.yml
-a----        11/27/2023   9:31 PM           1663 git_push.sh
-a----        11/27/2023   9:31 PM         351139 README.md
-a----        11/27/2023   9:31 PM             96 requirements.txt
-a----        11/27/2023   9:31 PM           1811 setup.py
-a----        11/27/2023   9:31 PM             69 test-requirements.txt
-a----        11/27/2023   9:31 PM            149 tox.ini
\n

We have code, we have tests, and even documentation. Here is a usage example from\none of the auto-generated docs:

\n
from __future__ import print_function
import time
import swagger_client
from swagger_client.rest import ApiException
from pprint import pprint

# Configure OAuth2 access token for authorization: oauth2_client_credentials
configuration = swagger_client.Configuration()
configuration.access_token = 'YOUR_ACCESS_TOKEN'

# create an instance of the API class
api_instance = swagger_client.AcademicWeeksApi(swagger_client.ApiClient(configuration))
id = 'id_example' # str | A resource identifier that uniquely identifies the resource.
if_match = 'if_match_example' # str | The ETag header value used to prevent the DELETE from removing a resource modified by another consumer. (optional)

try:
    # Deletes an existing resource using the resource identifier.
    api_instance.delete_academic_week_by_id(id, if_match=if_match)
except ApiException as e:
    print(\"Exception when calling AcademicWeeksApi->delete_academic_week_by_id: %s\\n\" % e)
\n

Converting to Poetry​

\n

I like to use Poetry for managing Python packages\ninstead of Pip, Conda, Tox, etc. Converting the requirements.txt file for use\nin Poetry is quite easy with this PowerShell command (hat\ntip):

\n
Push-Location edfi-client
poetry init --name edfi-client -l Apache-2.0
@(cat requirements.txt) | %{&poetry add $_.replace(' ','')}
Pop-Location
\n

(The default requirements.txt file has some unexpected spaces; the replace\ncommand above strips those out).

\n

Missing Token Generation​

\n

Note the line above with access_token = 'YOUR_ACCESS_TOKEN'. Swagger Codegen\nrequires you to bring your own token generation routine. We can build one using\nportions of the client library itself. The ODS/API supports the OAuth 2.0 client\ncredentials flow, which generates an bearer-style access token. A basic HTTP\nrequest for authentication looks like this:

\n
POST /v7.1/api/oauth/token HTTP/1.1
Host: api.ed-fi.org
Content-Type: application/x-www-form-urlencoded
Accept: application/json

grant_type=client_credentials&client_id=YOUR CLIENT ID&client_secret=YOUR CLIENT SECRET
\n

There are some variations in how these parameters can be passed, but this may be\nthe most common / universal format, and this is what we will implement here.

\n

Generated tokens are only good for so long; they expire. When a token expires,\nit would be nice if we could recognize that and automatically call for a new\none, instead of encountering an error. The generated code does not support token\nrefresh, and does not have an obvious hook for how to do so. For a very clean\ndeveloper experience, the authentication and refresh mechanisms would be built\ninto the ApiClient class created by Swagger Codegen. But be warned: if you\nrerun the generator, it will overwrite your customizations.

\n

Someone with deeper Python expertise can probably come up with multiple ways to\napproach the refresh problem. This sample code handles token refresh very\ncrudely, requiring the user of the code to detect the problem and try to\nre-authenticate. Perhaps a Context\nManager\nimplementation would help here.

\n

Copy the following source code and paste it into a file called\ntoken_manager.py inside the edfi-client/swagger_client directory.

\n
import json
from datetime import datetime, timedelta

from swagger_client.rest import ApiException
from swagger_client.configuration import Configuration
from swagger_client.api_client import ApiClient

class TokenManager(object):

    \"\"\"
    Creates a new instance of the TokenManager.

    Parameters
    ---------
    token_url: str
        The token URL for the Ed-Fi API.
    configuration: Configuration
        A list dictionary of configuration options for the RESTClientObject.
        Must study the RESTClientObject constructor carefully to understand the
        available options.
    \"\"\"
    def __init__(self, token_url: str, configuration: Configuration) -> None:
        assert token_url is not None
        assert token_url.strip() != \"\"

        self.token_url: str = token_url
        self.configuration: Configuration = configuration
        self.client: ApiClient = ApiClient(self.configuration)
        self.expires_at = datetime.now()

    def _authenticate(self) -> None:
        post_params = {
            \"grant_type\": \"client_credentials\",
            \"client_id\": self.configuration.username,
            \"client_secret\": self.configuration.password
        }
        headers = {
            \"Content-Type\": \"application/x-www-form-urlencoded\"
        }

        token_response = self.client.request(\"POST\", self.token_url, headers=headers, post_params=post_params)

        data = json.loads(token_response.data)
        self.expires_at = datetime.now() + timedelta(seconds=data[\"expires_in\"])
        self.configuration.access_token = data[\"access_token\"]

    \"\"\"
    Sends a token request and creates an ApiClient containing the returned access token.

    Returns
    ------
    ApiClient
        an ApiClient instance that has already been authenticated.
    \"\"\"
    def create_authenticated_client(self) -> ApiClient:

        self._authenticate()

        return self.client

    \"\"\"
    Re-authenticates if the token has expired.
    \"\"\"
    def refresh(self) -> None:
        if datetime.now() > self.expires_at:
            self._authenticate()
        else:
            raise ApiException(\"Token is not expired; authentication failure may be a configuration problem.\")
\n

Demonstration​

\n

The following snippet demonstrates use the token manager with a simple token\nrefresh mechanism. Note that this tries to delete an object that does not exist,\ntherefore you should expect it to raise an exception with a 404 NOT FOUND\nmessage.

\n
from swagger_client.configuration import Configuration
from swagger_client.token_manager import TokenManager
from swagger_client.api import AcademicWeeksApi
from swagger_client.rest import ApiException

BASE_URL = \"https://api.ed-fi.org/v7.1/api\"

config = Configuration()
config.username = \"RvcohKz9zHI4\"
config.password = \"E1iEFusaNf81xzCxwHfbolkC\"
config.host = f\"{BASE_URL}/data/v3/\"
config.debug = True

tm = TokenManager(f\"{BASE_URL}/oauth/token\", config)
api_client = tm.create_authenticated_client()

api_instance = AcademicWeeksApi(api_client)

try:
    api_instance.delete_academic_week_by_id(\"bogus\")
except ApiException as ae:
    if ae.status == 401:
        tm.refresh()
        api_instance.delete_academic_week_by_id(\"bogus\")
    else:
        raise
", "url": "https://blog.safnet.com/2023/12/08/edfi-client-from-swagger", "title": "Ed-Fi Client Generation in Python with Swagger CLI", "summary": "Motivation", "date_modified": "2023-12-08T00:00:00.000Z", "tags": [ "ed-fi", "programming" ] }, { "id": "https://blog.safnet.com/2023/08/10/responsible-data-use", "content_html": "

A strange thing about my job is that, although we're all about supporting K-12\neducation data interoperability, we don't actually work with any K-12 data. We\nbuild software, and others use it to collect data from disparate data sources\ninto a single, unified, and standardized data set. But that does't stop me from\nthinking about how data should be used.

\n

On a flight out to the #STATSDC2023 conference hosted by the National Center for\nEducational Statistics (my first time at this event), I finally wrote down my\npersonal principles for ethical / responsible use of data and AI. Many have\nwritten about responsible use of data; there is nothing ground breaking here.\nYet it feels meaningful, even if only for myself, to acknowledge \"out loud\" the\nvalues and principles that I wish to hold myself accountable for whenever I do\nuse data, encourage others to make use of data, allow my own data to be used,\netc.

\n

☝Note First draft, 2023-08-10

\n

Virtues and Values​

\n

As a person of faith, I tend to think of virtues before I think of values.\nVirtues are the spiritual foundations, and values are culturally-relevant\nextensions of those virtues. While there are countless variations on virtues and\nvalues, those below can serve as guiding lights for the principles that follow.

\n
\n

\"Truthfulness is the foundation of all human virtues.\" `Abdu'l-Bahá

\n
\n
VirtueValue Correlate(s)
truthfulnesstransparency, honesty, truth-seeking
justiceequity, independence, consensus
nobilityhuman dignity, recognition of aspirations, privacy
\n

Guiding Principles​

\n

Admittedly this is a hodge-podge list of principles, with no attempt to be\nsystematic at this time.

\n

Data Source Transparency: Retain as much information about data sources as\npossible. Vet them for integrity, completeness, appropriateness, and manner of\ncollection. Only use ethically sourced information. Insure personal data usage\ncorresponds to subject's intent. Give credit where credit is due.

\n

Example: avoid (shady) data aggregator / brokers.

\n

Algorithmic Transparency: Prefer open algorithms. Look for, or conduct,\naudits to assess accuracy, applicability to the desired use case(s), and\npotential for biases. Look for explainability and ensure there is a reasonable\nhuman appeals process. Does a great deep learning model really outweigh having\na good and explainable decision tree or a regression model?

\n

Example: Do AI and cheat detection systems have low false positives? Are there\nclear appeals mechanisms? Have they been cross-validated against different\nsubgroups to ensure they do not introduce biases? Can you explain why a person's\napplication is rejected or a promotion has been offered to one and not another\nwhen AI is involved in the process?

\n

Side note: Does use of generative AI really matter in the given context?

\n

Equity: Do not be content with doing no harm, rather seek to uplift\nmarginalized perspectives and peoples. Consider institutional biases that may\ninfluence raw data (over and under representation). Consider environmental\nfactors that may impact some populations, independent of or in concert with\nother demographic factors. Be mindful of accidental proxies.

\n

Examples: Account for ethnic / racial disparities in policing, gender disparities in\nsurveying, and geographic impacts such as longer bus rides to schools (\"all the\nXYZ kids have poor attendance\" might be due to late buses, or working parents,\nnothing to do with ethnicity).

\n

Don't feel virtuous by avoiding use of race while at the same time analyzing\nbased on income, if income and race are highly correlated in a particular area.

\n

Statistical Honesty: Use appropriate sample sizes and/or statistical tests\nto confirm assumptions. Cross validate results. Use large iterations when\nbootstrapping and repeat to insure stability of results.

\n

Privacy: Practice appropriate anonymization and apply \"need to know\" (least\nprivilege) restrictions. Watch out for small sample sizes that can\nunintentionally reveal identity. Use the least personal data available and\nrelevant for the analysis.

\n

Examples: Imagine an analysis that predicts a certain outcome for a target\ndemographic, and the dataset only contains a few people from that demographic.\nApply a filter by zip code. How hard is it to guess the person / household?

\n

Can you use census block instead of street address? Zip code instead of census\nblock? Age range instead of birth year?

\n

Consensus and Review: Seek review of data, algorithms, and outcomes from\nother experts and/or affected parties. Work to achieve consensus on techniques\nand communication. Start early to avoid the trap of stubbornly clinging to a\nmisdirected idea.

\n

Destiny: as a matter of justice, of nobility, and of truthfulness: be on\nguard for predictions becoming destiny.

\n

Example: Whether policing on the street or teachers disciplining students, a\nprediction that one group will have more criminal / behavioral incidents easily\nleads to paying more attention to that group, ignoring other groups, and thus\ndetecting (or \"inventing\") more ill behavior in that group - thus reinforcing\nthe original prediction.

\n
\n

Sources of Inspiration​

\n

Speaking of giving credit where credit is due, here are some sources that have\nhelped shape my thinking on this topic:

\n\n

And articles, asides, and conversations that are too numerous to remember or\ncite.

\n

Also see references in:

\n", "url": "https://blog.safnet.com/2023/08/10/responsible-data-use", "title": "Thoughts on Responsible Data Use", "summary": "A strange thing about my job is that, although we're all about supporting K-12", "date_modified": "2023-08-10T00:00:00.000Z", "tags": [ "discourse", "justice", "inclusion", "technology", "data-and-analytics" ] }, { "id": "https://blog.safnet.com/2023/05/08/meadowlark-cascading-updates", "content_html": "

Design document describing an event-driven approach for offline cascading\nupdates in the Ed-Fi Meadowlark project's document storage system: Design for\nOffline Cascading\nUpdates

", "url": "https://blog.safnet.com/2023/05/08/meadowlark-cascading-updates", "title": "Project Meadowlark: Design for Offline Cascading Updates", "summary": "Design document describing an event-driven approach for offline cascading", "date_modified": "2023-08-07T00:00:00.000Z", "tags": [ "ed-fi", "programming", "architecture" ] } ] }