85 posts tagged with "programming"

In Unpopular Opinion: It’s harder than ever to be a good software engineer, author Juraj Malencia opines that "good engineering" equates to "bring[ing] maximum value in achieving a goal." This is a comforting thought for those of us periodically wondering if we're still good software or data engineers, though we might not be up-to-date on the latest hype.

Pointing out that "engineering does not equal programming", implying that we can bring great "engineering" (solutioning) value without writing a line of code, he presents a modified version of a venn diagram on how people in various roles spend their time between programming, alignment, people, and "other". I was bemused to note I currently sit squarely in the category labeled "beware" 🤨. A position that may continue throughout this year, but will need to change over time.

Motivation​

The Ed-Fi ODS/API is a REST API that support interoperability of student data systems. The API definition, via the Ed-Fi Data Standard, is extensible: many large-scale or specialized implementations add their own local use cases that are not supported out of the box by the Ed-Fi Data Standard (Extensions). Furthermore, the Data Standard receives regular updates; sometimes these are merely additive, and from time to time there are breaking changes. These factors make it impossible to create a one-size fits all client library.

But, not all is lost: the ODS/API exposes its API definition using OpenAPI, and we can use Swagger Codegen to build a client library based on the target installation's data model / API spec. The basic process of creating a C# code library (SDK) is described in Ed-Fi documentation at Using Code Generation to Create an SDK (Note: this link is for ODS/API 7.1, but the instructions are essentially the same for all versions).

Design document describing an event-driven approach for offline cascading updates in the Ed-Fi Meadowlark project's document storage system: Design for Offline Cascading Updates

Design document describing approaches and appropriateness of in-memory and distributed caching for improving the performance of referential integrity checks in the Ed-Fi Meadowlark API: Resource Caching for Performance Improvements

Full write-up of experiments in MongoDB connection string tuning with Ed-Fi Project Meadowlark: RND-380: MongoDB Connection Pooling

As education agencies increasingly look for cost-savings and scalability benefits through adoption of hybrid or cloud-only IT infrastructures, it is natural to ask: what should an ideal Ed-Fi installation look like in the cloud?

Originally posted on www.ed-fi.org

The Ed-Fi Tech Congress in Phoenix, of April 2018, was a sink or swim moment for me, as I had just started working for the Ed-Fi Alliance. Among the first people I met was a representative from one of the big technology companies. The conversation quickly turned to the question of how to deal with data when the vendor would not send it directly into the Ed-Fi ODS/API. He asked me, "why not just put it in a data lake?" To which I had no reply. Nearly four years later, at last I can give a reasonable reply.

Continue reading on wwww.ed-fi.org...

Prompted by a class I'm taking, I decided to try running Python from Windows Subsystem for Linux (WSL; actually, WSL2 to be specific). Installing Python in Ubuntu on Windows was relatively easy, though I did run into a couple of little problems with running poetry. Bigger challenge: running graphical user interfaces (GUIs) from WSL. Here are some quick notes from my experience.

Goal: setup PowerShell Core and .NET for development in Ubuntu running in Windows subsystem for Linux (WSL). And a few other tools.

Motivation: porting PowerShell scripts for .NET development on Linux, thus enabling more programmers to develop on a certain codebase and enabling use of Linux-based containers for continuous integration and testing.

Author Neal Stephenson, in his essay "In the Beginning... Was the Command Line," memorably compares our graphical user interfaces to Disney theme parks: "It seems as if a hell of a lot might be being glossed over, as if Disney World might be putting one over on us, and possibly getting away with all kinds of buried assumptions and muddled thinking. And this is precisely the same as what is lost in the transition from the command line interface to the GUI. (p52)

With new programmers whose experience has been entirely mediated through an

IDE like Visual Studio or Eclipse, I have sometimes wondered if they are understanding the "buried assumptions" and suffering from "muddled thinking" due to their lack of understanding of the basic command line operations that underlie the automation provided in the IDE. I still recall when I was that young developer, who had started with nothing but the command line, and realized that Visual Studio had crippled my ability to know how to build and test .NET Framework solutions (setting up an automated build process in Cruise Control helped cure me of that).