164 posts tagged with "tech"

Data engineering gurus Joe Reis and Matthew Housley once again led a closing town hall at Data Day Texas. Rather than opining from the front, they turned the session over to the wisdom of the crowd. Housley seeded the conversation with a single question – “what is the elephant in the room?” – and the room was ready with an answer: AI. In particular: what is AI going to do to my job?

Given a room full of strangers, some participants were remarkably open about their fears. Perhaps knowing that the audience is composed of fellow data geeks helped to establish a sense of vulnerability. These fears were being expressed by the people who, in theory, should be the ones developing expertise in using AI tooling. But that’s how disruptive the technologies may be: even the data experts are uncertain and afraid.

Notes from three talks on data cataloging at the recent Data Day Texas 2025

• The Meta Grid
• Investing in Semantics and Knowledge
• We Are All Librarians

Following up on Grudgingly Accepting AI Coding Assistants, I have a small win from my first experiment with GitHub Copilot: a Python script for converting portions of an Open API specification file to a CSV. There is nothing revolutionary in this small win; countless programmers have doubtlessly done much more. But a reluctant engineer has to start somewhere.

As a software engineering director building open source products, I have prohibited my teams from using AI coding assistants due to concerns about intellectual property and questions about the risks and real world effectiveness of AI coding assistants. It is now time to allow and even encourage AI coding assistants, with guardrails.

In reaction to all of the "agile is dead" articles, I am cleaning up old posts about Agile, re-reading them, contemplating lessons learned but forgotten, and asking myself if some practices have outlived their usefulness. That is the spirit of agility: the interplay of action and reflection.

"Agile" is not a silver bullet for improving software productivity, reliability, and simplicity. But "Agile" continues to give us tools that can foster improved software engineering.

The Agile Manifesto was a distillation of certain trends that the authors had noticed in their successful projects. On balance, they ring true to my twenty-five year career in software.

It did not offer guarantees and it did not offer to solve world hunger. Through prescriptive frameworks such as Extreme Programming and Scrum, and common practices such as development of story cards and short cycles (aka sprints), the Agile "revolution" broke us free from the confines of gigantic requirements and design documents that were always at least slightly wrong, and frequently very difficult to change. It helped us embrace the uncertainty of software development, empowering us to find our way out of that wrongness more quickly and productively.

A strange thing about my job is that, although we're all about supporting K-12 education data interoperability, we don't actually work with any K-12 data. We build software, and others use it to collect data from disparate data sources into a single, unified, and standardized data set. But that does't stop me from thinking about how data should be used.

On a flight out to the #STATSDC2023 conference hosted by the National Center for Educational Statistics (my first time at this event), I finally wrote down my personal principles for ethical / responsible use of data and AI. Many have written about responsible use of data; there is nothing ground breaking here. Yet it feels meaningful, even if only for myself, to acknowledge "out loud" the values and principles that I wish to hold myself accountable for whenever I do use data, encourage others to make use of data, allow my own data to be used, etc.

Prompted by a class I'm taking, I decided to try running Python from Windows Subsystem for Linux (WSL; actually, WSL2 to be specific). Installing Python in Ubuntu on Windows was relatively easy, though I did run into a couple of little problems with running poetry. Bigger challenge: running graphical user interfaces (GUIs) from WSL. Here are some quick notes from my experience.

Goal: setup PowerShell Core and .NET for development in Ubuntu running in Windows subsystem for Linux (WSL). And a few other tools.

Motivation: porting PowerShell scripts for .NET development on Linux, thus enabling more programmers to develop on a certain codebase and enabling use of Linux-based containers for continuous integration and testing.

Author Neal Stephenson, in his essay "In the Beginning... Was the Command Line," memorably compares our graphical user interfaces to Disney theme parks: "It seems as if a hell of a lot might be being glossed over, as if Disney World might be putting one over on us, and possibly getting away with all kinds of buried assumptions and muddled thinking. And this is precisely the same as what is lost in the transition from the command line interface to the GUI. (p52)

With new programmers whose experience has been entirely mediated through an

IDE like Visual Studio or Eclipse, I have sometimes wondered if they are understanding the "buried assumptions" and suffering from "muddled thinking" due to their lack of understanding of the basic command line operations that underlie the automation provided in the IDE. I still recall when I was that young developer, who had started with nothing but the command line, and realized that Visual Studio had crippled my ability to know how to build and test .NET Framework solutions (setting up an automated build process in Cruise Control helped cure me of that).

"Infrastructure as Code", or IaC if you prefer TLAs, is the practice of configuring infrastructure components in text files instead of clicking around in a user interface. Last year I wrote a few detailed articles on IaC with TeamCity (1, 2, 3). Today I want take a step back and briefly address the topic more broadly, particularly with respect to continuous integration (CI) and delivery (CD): the process of automating software compilation, testing, quality checks, packaging, deployment, and more.