Home on Steven Engelhardt

Wednesday 2025-05-28 Assorted Links

Wed, 28 May 2025 00:00:00 +0000

Assorted links for Wednesday, May 28:

Unlocking the Power of Regex in SQL Server
Announcing the General Availability (GA) of JSON data type & JSON aggregates…

We are excited to announce the General Availability (GA) of the native JSON data type and JSON aggregates – JSON_OBJECTAGG & JSON_ARRAYAGG. You can use the JSON data type and JSON aggregates to integrate and work with JSON documents more efficiently in the database. This functionality is available in Azure SQL Database, Azure SQL Managed Instance with the Always-up-to-date update policy.
C# 14 – Exploring extension members
Evaluating content safety in your .NET AI applications

We are excited to announce the addition of the Microsoft.Extensions.AI.Evaluation.Safety package to the Microsoft.Extensions.AI.Evaluation libraries! This new package provides evaluators that help you detect harmful or sensitive content — such as hate speech, violence, copyrighted material, insecure code, and more — within AI-generated content in your Intelligent Applications.
.NET Aspire 9.3 is here and enhanced with GitHub Copilot!

Tuesday 2025-05-27 Assorted Links

Tue, 27 May 2025 00:00:00 +0000

Assorted links for Tuesday, May 27:

Fast, Accurate, and Affordable: Sharded DiskANN for Multitenant Vector Search in Azure Cosmos DB

Azure Cosmos DB now includes Sharded DiskANN, a powerful capability that optimized for large-scale multitenant apps by splitting a DiskANN index into smaller, more performant pieces.
General Availability for Data API in vCore-based Azure Cosmos DB for MongoDB

The Data API is a RESTful HTTPS interface that enables developers to interact with their MongoDB data hosted in vCore-based Azure Cosmos DB directly from their applications—no drivers or complex query logic required. It offers a simple, efficient way to connect your MongoDB data with web apps, mobile apps, and tools like Power BI.
Elevating Azure Cosmos DB Resilience with Per Partition Automatic Failover

Today, we’re excited to announce the preview of Per Partition Automatic Failover (PPAF) for Azure Cosmos DB, a significant improvement to our single-region write accounts that boosts availability and resilience.
Boost Query Performance with Global Secondary Indexes in Azure Cosmos DB

Global secondary indexes for Azure Cosmos DB—now in Public Preview—make it easier to query data efficiently, especially as your datasets grow.
New Generally Available and Preview Search Capabilities in Azure Cosmos DB for NoSQL

Monday 2025-05-26 Assorted Links

Mon, 26 May 2025 00:00:00 +0000

Assorted links for Monday, May 26:

How Meta understands data at scale
- Managing and understanding large-scale data ecosystems is a significant challenge for many organizations, requiring innovative solutions to efficiently safeguard user data. Meta’s vast and diverse systems make it particularly challenging to comprehend its structure, meaning, and context at scale.
- To address these challenges, we made substantial investments in advanced data understanding technologies, as part of our Privacy Aware Infrastructure (PAI). Specifically, we have adopted a “shift-left” approach, integrating data schematization and annotations early in the product development process. We also created a universal privacy taxonomy, a standardized framework providing a common semantic vocabulary for data privacy management across Meta’s products that ensures quality data understanding and provides developers with reusable and efficient compliance tooling.
Efficiently and Elegantly Modeling Embeddings in Azure SQL and SQL Server

Storing and querying text embeddings in a database it might seem challenging, but with the right schema design, it’s not only possible, it’s powerful. Whether you’re building AI-powered search, semantic filtering, or recommendation features, embeddings, and thus vectors, are now a first-class data type. So how do you model them well inside a database like SQL Server and Azure SQL?
Avoid T-SQL anti-patterns with the free T-SQL analysis tool

T-SQL Analyzer is a free, open-source, cross platform command line tool for identifying, and reporting the presence of anti-patterns and design issues in SQL Server T-SQL scripts.
Surprising Scalability of Multitenancy

Roughly speaking, the cost of a system scales with its (short-term1) peak traffic, but for most applications the value the system generates scales with the (long-term) average traffic.

The gap between “paying for peak” and “earning on average” is critical to understand how the economics of large-scale cloud systems differ from traditional single-tenant systems.
FAST ‘23 - Building and Operating a Pretty Big Storage System (My Adventures in Amazon S3)

Monday 2025-05-05 Assorted Links

Mon, 05 May 2025 00:00:00 +0000

Assorted links for Monday, May 5:

How we ended up rewriting NuGet Restore in .NET 9

This is the story of how team members across NuGet, Visual Studio, and .NET embarked on a journey to fully rewrite the NuGet Restore algorithm to achieve break-through scale and performance. Written from the perspective of several team members, this entry provides a deep dive into the internals of NuGet, as well as strategies to identify and address performance issues. We hope that you enjoy it!
How Netflix Accurately Attributes eBPF Flow Logs

In a previous blog post, we described how Netflix uses eBPF to capture TCP flow logs at scale for enhanced cloud network insights. In this post, we delve deeper into how Netflix solved a core problem: accurately attributing flow IP addresses to workload identities.
Going beyond singleton, scoped, and transient lifetimes—tenant, pooled, and drifter

In this post I first briefly describe the standard lifetimes available in the .NET DI container. I then briefly describe the three hypothetical lifetimes described in the podcast. Finally, I show how you could implement one of these lifetimes in practice. In the next post I show a possible implementation for the remaining lifetime.
Multi-tenancy in ASP.NET Core 8 - Dependency Injection & Tenant Specific Services

This post discusses how we can have tenant specific services in a multi-tenant ASP.NET Core 8 application.
Enhancing Search Capabilities in SQL Server and Azure SQL with Hybrid Search and RRF Re-Ranking

In today’s data-driven world, delivering precise and contextually relevant search results is critical. SQL Server and Azure SQL Database now enable this through Hybrid Search—a technique that combines traditional full-text search with modern vector similarity search. This allows developers to build intelligent, AI-powered search experiences directly inside the database engine.

Wednesday 2025-04-09 Assorted Links

Wed, 09 Apr 2025 00:00:00 +0000

Assorted links for Wednesday, April 9:

A large scale analysis of hundreds of in-memory cache clusters at Twitter

In this work, we significantly further the understanding of real-world cache workloads by collecting production traces from 153 in-memory cache clusters at Twitter, sifting through over 80 TB of data, and sometimes interpreting the workloads in the context of the business logic behind them.
Durability: Linux File APIs

[Y]ou should assume your data is corrupt between when a write is issued until after a flush or force unit access write completes. However, most programs use system calls to write data. This article looks at the guarantees provided by the Linux file APIs. It seems like this should be simple: a program calls write() and after it completes, the data is durable. However, write() only copies data from the application into the kernel’s cache in memory. To force the data to be durable you need to use some additional mechanism.
How Netflix Scales its API with GraphQL Federation (Part 1)

As we’ve grown the number of developers and increased our domain complexity, developing the API aggregation layer has become increasingly harder.

In order to address this rising problem, we’ve developed a federated GraphQL platform to power the API layer.
Traefik: canary deployments with weighted load balancing

Traefik is The Cloud Native Edge Router yet another reverse proxy and load balancer. Omitting all the Cloud Native buzzwords, what really makes Traefik different from Nginx, HAProxy, and alike is the automatic and dynamic configurability it provides out of the box. And the most prominent part of it is probably its ability to do automatic service discovery.
Rust vs Go in 2025

Which is better, Rust or Go—and does that question even make sense? Which language should you choose for your next project in 2025, and why? How does Rust compare with Go in areas like performance, simplicity, safety, features, scale, and concurrency?

Tuesday 2025-04-08 Assorted Links

Tue, 08 Apr 2025 00:00:00 +0000

Assorted links for Tuesday, April 8:

Building a ubiquitous shared infrastructure using Twine

Twine is our homegrown cluster management system, which has been running in production for the past decade. A cluster management system allocates workloads to machines and manages the life cycle of machines, containers, and workloads. Kubernetes is a prominent example of an open source cluster management system. Twine has helped convert our infrastructure from a collection of siloed pools of customized machines dedicated to individual workloads to a large-scale ubiquitous shared infrastructure in which any machine can run any workload.
Ensuring data reaches disk

The purpose of this document is to describe the path data takes from the application down to the storage, concentrating on places where data is buffered, and to then provide best practices for ensuring data is committed to stable storage so it is not lost along the way in the case of an adverse event. The main focus is on the C programming language, though the system calls mentioned should translate fairly easily to most other languages.
Revolutionizing Money Movements at Scale with Strong Data Consistency

As one of the underlying engines, Uber Money fulfills some of the most important aspects of people’s engagement in the Uber experience. A system like this should not only be robust, but should also be highly available with zero-tolerance to downtime, after our success mantra: “To collect and disburse on-time, accurately and in-compliance”.

While we expand to multiple lines of businesses, and strategize the next best, the engineers in Uber Money also thrive on building the next generation’s Payments Platform which extends Uber’s growth. In this blog, we introduce you to this platform and provide insights into our learnings. This includes migrating hundreds of millions customers between two asynchronous systems while maintaining data-consistency with a goal of zero impact on our users.
How 30 Lines of Code Blew Up a 27-Ton Generator

A secret experiment in 2007 proved that hackers could devastate power grid equipment beyond repair—with a file no bigger than a GIF.
Avoiding overload in distributed systems by putting the smaller service in control

Within AWS, a common pattern is to split the system into services that are responsible for executing customer requests (the data plane), and services that are responsible for managing and vending customer configuration (the control plane). In this article, I discuss a number of different ways the data plane and the control plane interact with each other to avoid system overload. In many of these architectures the larger data plane fleet calls the smaller control plane fleet, but I also want to share the success we’ve had at Amazon when we put the smaller fleet in control.

Monday 2025-04-07 Assorted Links

Mon, 07 Apr 2025 00:00:00 +0000

Assorted links for Monday, April 7:

The Lost Xperf Documentation–Disk Usage

As I’ve lamented previously, the documentation for xperf (Windows Performance Toolkit) is a bit light. The names of the columns in the summary tables can be exquisitely subtle, and I have never found any documentation for them. But, I’ve talked to the xperf authors, and I’ve used xperf a lot, and I’ve done some experiments, and here I share some more results, this time for the Disk Usage summary table.
The macro problem with microservices

In just 20 years, software engineering has shifted from architecting monoliths with a single database and centralized state to microservices where everything is distributed across multiple containers, servers, data centers, and even continents. Distributing things solves scaling concerns, but introduces a whole new world of problems, many of which were previously solved by monoliths.
Learnings From Two Years of Kubernetes in Production
FioSynth: A representative I/O benchmark and data visualizer for data center workloads

FioSynth is a benchmark tool used to automate the execution of storage workload suites and to parse results. It contains a base set of block level storage workloads, synthesized from production I/O traces, that simulate a diverse range of Facebook production services. It is useful for predicting how a storage device will perform in realistic production environments and for assisting with performance tuning.
Azure Container Registry Adds Teleportation

Project Teleport removes the cost of download and decompression by SMB mounting pre-expanded layers from the Azure Container Registry to Teleport enabled Azure container hosts.

Friday 2025-04-04 Assorted Links

Fri, 04 Apr 2025 00:00:00 +0000

Assorted links for Friday, April 4:

Quake’s 3-D Engine: The Big Picture
Reasoning about colors

In July 2020 I went on a color-scheme vision quest. This led to some research on various color spaces and their utility, some investigation into the styling guidelines outlined by the base16 project, and the color utilities that ship within the GNU Emacs text editor. This article will be a whirlwind tour of things you can do to individual colors, and at the end how I put these blocks together.
Modern storage is plenty fast. It is the APIs that are bad.

In this article I will demonstrate that while hardware changed dramatically over the past decade, software APIs have not, or at least not enough. Riddled with memory copies, memory allocations, overly optimistic read ahead caching and all sorts of expensive operations, legacy APIs prevent us from making the most of our modern devices.
How io_uring and eBPF Will Revolutionize Programming in Linux

[eBPF and io_uring] may look evolutionary, but they are revolutionary in the sense that they will — we bet — completely change the way applications work with and think about the Linux Kernel.
An ex-Googler’s guide to dev tools

I thought it would be helpful to write a guide to dev tools outside of Google for the ex-Googler, written with an eye toward pragmatism and practicality. No doubt many ex-Googlers wish they could simply clone the Google internal environment to their new company, but you can’t boil the ocean. Here is my take on where you should start and a general path I think ex-Googlers can take to find the tools that will make them - and their new teams - as productive as possible.

Thursday 2025-04-03 Assorted Links

Thu, 03 Apr 2025 00:00:00 +0000

Assorted links for Thursday, April 3:

A Critique of Snapshot Isolation

There have been recent attempts to enrich large-scale data stores, such as HBase and BigTable, with transactional support. Not surprisingly, inspired by traditional database management systems, serializability is usually compromised for the benefit of efficiency. For example, Google Percolator, implements lock-based snapshot isolation on top of BigTable. We show in this paper that this compromise is not necessary in lock-free implementations of transactional support. We introduce write-snapshot isolation, a novel isolation level that has a performance comparable with that of snapshot isolation, and yet provides serializability.
Making Snapshot Isolation Serializable

This article develops a theory that characterizes when nonserializable executions of applications can occur under [snapshot isolation].
Weak Consistency: A Generalized Theory and Optimistic Implementations for Distributed Transactions

This thesis presents the first implementation-independent specifications of existing ANSI isolation levels and a number of levels that are widely used in commercial systems, e.g., Cursor Stability, Snapshot Isolation. It also specifies a variety of guarantees for predicate-based operations in an implementation-independent manner. Two new levels are defined that provide useful consistency guarantees to application writers; one is the weakest level that ensures consistent reads, while the other captures some useful consistency properties provided by pessimistic implementations.
What Does Write Skew Look Like?

This post is about gaining intuition for Write Skew, and, by extension, Snapshot Isolation. Snapshot Isolation is billed as a transaction isolation level that offers a good mix between performance and correctness, but the precise meaning of “correctness” here is often vague. In this post I want to break down and capture exactly when the thing called “write skew” can happen.
Constructing human-grade parsers

Here are some observations on how parsers can be constructed in a way that makes it easier to recover from parse errors, produce multiple diagnostics in one pass, and provide partial results for further analysis even in the face of errors, providing a better experience for user-driven command line tools and interactive environments.

Wednesday 2025-04-02 Assorted Links

Wed, 02 Apr 2025 00:00:00 +0000

Assorted links for Wednesday, April 2:

Build systems à la carte: Theory and practice

Build systems are awesome, terrifying – and unloved. They are used by every developer around the world, but are rarely the object of study. In this paper, we offer a systematic, and executable, framework for developing and comparing build systems, viewing them as related points in a landscape rather than as isolated phenomena. By teasing apart existing build systems, we can recombine their components, allowing us to prototype new build systems with desired properties.
Erasure Coding in Windows Azure Storage

In this paper we introduce a new set of codes for erasure coding called Local Reconstruction Codes (LRC). LRC reduces the number of erasure coding fragments that need to be read when reconstructing data fragments that are offline, while still keeping the storage overhead low.
Reproducible Builds

Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code.
Getting to Deterministic Builds on Windows

This is a set of notes on getting to deterministic builds in C, C++ and Rust on Windows.
Our Software Dependency Problem

Software dependencies carry with them serious risks that are too often overlooked. The shift to easy, fine-grained software reuse has happened so quickly that we do not yet understand the best practices for choosing and using dependencies effectively, or even for deciding when they are appropriate and when not. My purpose in writing this article is to raise awareness of the risks and encourage more investigation of solutions.

Tuesday 2025-04-01 Assorted Links

Tue, 01 Apr 2025 00:00:00 +0000

Assorted links for Tuesday, April 1:

Cloudflare turns AI against itself with endless maze of irrelevant facts

On Wednesday, web infrastructure provider Cloudflare announced a new feature called “AI Labyrinth” that aims to combat unauthorized AI data scraping by serving fake AI-generated content to bots. The tool will attempt to thwart AI companies that crawl websites without permission to collect training data for large language models that power AI assistants like ChatGPT.
War story: the hardest bug I ever debugged

All of a sudden, without any ostensible cause, Google Docs was flooded with errors. How it took me 2 days and a coworker to solve the hardest bug I ever debugged.
Introducing Styrolite: Building a Linux Container Runtime from Scratch

Edera Protect is a suite of offerings bridging the gap between modern cloud native computing and virtualization-based security techniques. To power this platform, we’ve built our own container runtime designed to operate as a microservice, allowing it to run containers in a fully programmatic way—similar to how the Kubernetes Container Runtime Interface (CRI) enables container management through microservices.
High-Performance PNG Codec

I would like to announce a new high-performance PNG codec, which is much faster than other available codecs written in C, C++, and other programming languages.
Image creation and testing with HashiCorp Packer and ServerSpec

Monday 2025-03-31 Assorted Links

Mon, 31 Mar 2025 00:00:00 +0000

Assorted links for Monday, March 31:

In S3 simplicity is table stakes

Today, on Pi Day (S3’s 19th birthday), I’m sharing a post from Andy Warfield, VP and Distinguished Engineer of S3. Andy takes us through S3’s evolution from simple object store to sophisticated data platform, illustrating how customer feedback has shaped every aspect of the service. It’s a fascinating look at how we maintain simplicity even as systems scale to handle hundreds of trillions of objects.
The xz attack shell script
Context is all you need: Better AI results with custom instructions

Earlier this month, we announced the general availability of custom instructions in Visual Studio Code. Custom instructions are how you give Copilot specific context about your team’s workflow, your particular style preferences, libraries the model may not know about, etc.

In this post we’ll dive into what custom instructions are, how you can use them today to drastically improve your results with GitHub Copilot, and even a brand new preview feature called “prompt files” that you can try today.
OpenAI adopts rival Anthropic’s standard for connecting AI models to data

In a post on X on Wednesday, OpenAI CEO Sam Altman said that OpenAI will add support for Anthropic’s Model Context Protocol, or MCP, across its products, including the desktop app for ChatGPT. MCP is an open source standard that helps AI models produce better, more relevant responses to certain queries.
Microsoft announces security AI agents to help overwhelmed humans

Microsoft’s six security agents will be available in preview next month, and are designed to do things like triage and process phishing and data loss alerts, prioritize critical incidents, and monitor for vulnerabilities.

Tuesday 2025-03-18 Assorted Links

Tue, 18 Mar 2025 00:00:00 +0000

Assorted links for Tuesday, March 18:

Researchers astonished by tool’s apparent success at revealing AI’s “hidden objectives”

In a new paper published Thursday titled “Auditing language models for hidden objectives,” Anthropic researchers described how custom AI models trained to deliberately conceal certain “motivations” from evaluators could still inadvertently reveal secrets, due to their ability to adopt different contextual roles they call “personas.” The researchers were initially astonished by how effectively some of their interpretability methods seemed to uncover these hidden training objectives, although the methods are still under research.
Why SNES hardware is running faster than expected—and why it’s a problem

After significant research and testing on dozens of actual SNES units, the TASBot team now thinks that a cheap ceramic resonator used in the system’s Audio Processing Unit (APU) is to blame for much of this inconsistency. While Nintendo’s own documentation says the APU should run at a consistent rate of 24.576 Mhz (and the associated Digital Signal Processor sample rate at a flat 32,000 Hz), in practice, that rate can vary just a bit based on heat, system age, and minor physical variations that develop in different console units over time.
The Defer Technical Specification: It Is Time

Time for me to write this blog post and prepare everyone for the implementation blitz that needs to happen to make defer a success for the C programming language.
Introducing support for SLNX, a new, simpler solution file format in the .NET CLI

Solution files have been a part of the .NET and Visual Studio experience for many years now, and they’ve had the same custom format the whole time. Recently, the Visual Studio solution team has begun previewing a new, XML-based solution file format called SLNX. Starting in .NET SDK 9.0.200, the dotnet CLI supports building and interacting with these files in the same way as it does with existing solution files.
Hello HybridCache! Streamlining Cache Management for ASP.NET Core Applications

HybridCache is a new .NET 9 library available via the Microsoft.Extensions.Caching.Hybrid package and is now generally available! HybridCache, named for its ability to leverage both in-memory and distributed caches like Redis, ensures that data storage and retrieval is optimized for performance and security, regardless of the scale or complexity of your application.

Monday 2025-03-17 Assorted Links

Mon, 17 Mar 2025 00:00:00 +0000

Assorted links for Monday, March 17:

Faster Go maps with Swiss Tables

Like sorting algorithms, hash table data structures continue to see improvements. In 2017, Sam Benzaquen, Alkis Evlogimenos, Matt Kulukundis, and Roman Perepelitsa at Google presented a new C++ hash table design, dubbed “Swiss Tables”. In 2018, their implementation was open sourced in the Abseil C++ library.

Go 1.24 includes a completely new implementation of the built-in map type, based on the Swiss Table design.
Harden-Runner detection: tj-actions/changed-files action is compromised

We are investigating a critical security incident involving the popular tj-actions/changed-files GitHub Action. We want to alert you immediately so that you can take prompt action. This post will be updated as new information becomes available.
Highlights from Git 2.49
1. Faster packing with name-hash v2
2. Backfill historical blobs in partial clones
Life Altering Postgresql Patterns: Many of these apply to all SQL, not just PostgreSQL
1. Use UUID primary keys
2. Give everything created_at and updated_at
3. On update restrict on delete restrict
4. Use schemas
5. Enum Tables
6. Name your tables singularly
7. Mechanically name join tables
8. Almost always soft delete
9. Represent statuses as a log
10. Mark special rows with a system_id
11. Use views sparingly
12. JSON Queries
cppmatch

A header-only C++ library that offers exceptionless error handling and type-safe enums, bringing Rust-inspired error propagation with the ? operator and the match operator to C++.

Friday 2025-03-14 Assorted Links

Fri, 14 Mar 2025 00:00:00 +0000

Assorted links for Friday, March 14:

Agentic AI is the New Web App, and Your AI Strategy Must Evolve

Two years into the generative AI revolution, the LLMs that power tools like ChatGPT and Claude have become startlingly powerful. However, according to Salesforce CEO Marc Benioff, they may be reaching their limits. Per Benioff, the next evolution is not necessarily more intelligent LLMs but autonomous AI agents that leverage LLMs to execute tasks independently.
Title Launch Observability at Netflix Scale
Performance of the Python 3.14 tail-call interpreter

About a month ago, the CPython project merged a new implementation strategy for their bytecode interpreter. The initial headline results were very impressive, showing a 10-15% performance improvement on average across a wide range of benchmarks across a variety of platforms.

Unfortunately, as I will document in this post, these impressive performance gains turned out to be primarily due to inadvertently working around a regression in LLVM 19.
Model Context Protocol

MCP is an open protocol that standardizes how applications provide context to LLMs. Think of MCP like a USB-C port for AI applications. Just as USB-C provides a standardized way to connect your devices to various peripherals and accessories, MCP provides a standardized way to connect AI models to different data sources and tools.
Traversal-resistant file APIs

A path traversal vulnerability arises when an attacker can trick a program into opening a file other than the one it intended. This post explains this class of vulnerability, some existing defenses against it, and describes how the new os.Root API added in Go 1.24 provides a simple and robust defense against unintentional path traversal.

Thursday 2025-03-13 Assorted Links

Thu, 13 Mar 2025 00:00:00 +0000

Assorted links for Thursday, March 13:

Instrumenting Apache Spark Structured Streaming jobs using OpenTelemetry

Monitoring Apache Spark structured streaming data workloads is challenging because the data is continuously processed as it arrives. Because of this always-on nature of stream processing, it is harder to troubleshoot problems during development and production without real-time metrics, alerting and dashboards. Traces complement metrics, and since Spark doesn’t include them by default, we integrate them using OpenTelemetry.
Protecting user data through source code analysis at scale

Meta’s Anti Scraping team focuses on preventing unauthorized scraping as part of our ongoing work to combat data misuse. In order to protect Meta’s changing codebase from scraping attacks, we have introduced static analysis tools into our workflow. These tools allow us to detect potential scraping vectors at scale across our Facebook, Instagram, and even parts of our Reality Labs codebases.
We’ve figured out the basics of a shape-shifting, T-1000-style material

Campàs and his team drew inspiration from processes called fluidization and convergent extension—mechanisms that cells in embryos use to coordinate their behavior when forming tissues and organs in a developing organism. The team built a robotic collective where each robotic unit behaved like an embryonic cell. As a collective, the robots behaved like a material that could change shape and switch between solid and liquid states, just like the T-1000.
Cross-Modal Retrieval: Why It Matters for Multimodal AI

With its ability to simultaneously process different data types (think text, image, audio, video and more), the continuing development of multimodal AI represents the next step that would help to further enhance a wide range of tools — including those for generative AI and autonomous agentic AI.
The Deployment Bottleneck No One Talks About

Most applications rely on cloud SDKs to connect to services like message brokers, queues, databases, APIs and more.

Rather than working directly with cloud SDKs, a better approach is to introduce a standardized layer between applications and cloud services. This allows developers to interact with essential resources without being tightly coupled to a specific provider’s SDKs. A framework like Dapr helps achieve this by providing a uniform API for interacting with cloud resources.

Wednesday 2025-03-12 Assorted Links

Wed, 12 Mar 2025 00:00:00 +0000

Assorted links for Wednesday, March 12:

Zen and the Art of Microcode Hacking

The root cause of the EntrySign vulnerability is that the AMD Zen microcode signature verification algorithm uses the CMAC function as a hash function; however, CMAC is a message authentication code and does not necessarily provide the same security guarantees as a cryptographic hash function.

The weakness of using CMAC as a hash function is that anyone who has the encryption key is able to observe the intermediate values of the encryption and calculate a way to “correct” the difference so that the final output remains the same, even if the inputs are completely different.
Thousands of websites hit by four backdoors in 3rd party JavaScript attack

While analyzing threats targeting WordPress frameworks, we found an attack where a single 3rd party JavaScript file was used to inject four separate backdoors into 1,000 compromised websites using cdn.csyndication[.]com/.

Creating four backdoors facilitates the attackers having multiple points of re-entry should one be detected and removed. A unique case we haven’t seen before. Which introduces another type of attack made possibly by abusing websites that don’t monitor 3rd party dependencies in the browser of their users.
How to debug code with GitHub Copilot

GitHub Copilot can streamline your debugging process by troubleshooting in your IDE, analyzing pull requests, and more, helping you tackle issues faster and more robustly.
Finding leaked passwords with AI: How we built Copilot secret scanning

Passwords are notoriously difficult to detect with conventional programming approaches. AI can help us find passwords better because it understands context. This blog post will explore the technical challenges we faced with building the feature and the novel and creative ways we solved them.
Monads

If you understand what a functor is, it should be easy to grasp the idea of a monad. It’s a functor you can flatten.

Tuesday 2025-03-11 Assorted Links

Tue, 11 Mar 2025 00:00:00 +0000

Assorted links for Tuesday, March 11:

How Generative AI Is Reshaping the SDLC

Amazon Q shows how GenAI is helping developers at all stages of code creation and delivery, said Srini Iragavarapu of AWS in this episode of Makers.
.NET AI Template Now Available in Preview

Want to get started with AI development, but not sure where to start? I’ve got a treat for you – we have a new AI Chat Web App template now in preview.
Rethinking System Architecture: The Rise of Distributed Intelligence with eBPF

With eBPF, we can process, filter, and act on data as it flows through the system — directly at the kernel level. This architecture approach flips the centralized model on its head by embedding decision-making directly into the system at the point where data is generated. This means that instead of forwarding vast amounts of raw data for centralized processing, we can use intelligent, kernel-embedded programs to analyze, process, and act on data exactly where it was generated in real-time. By doing this, eBPF enables a shift from centralized, reactive decision-making to distributed, proactive intelligence.
When AI Thinks It Will Lose, It Sometimes Cheats, Study Finds: This is the AI alignment problem, which has been explored extensively in science fiction.

When sensing defeat in a match against a skilled chess bot, [advanced AI models] don’t always concede, instead sometimes opting to cheat by hacking their opponent so that the bot automatically forfeits the game.
Unlock new possibilities for AI Evaluations for .NET

The Microsoft.Extensions.AI.Evaluations library is designed to simplify the integration of AI evaluation processes into your applications. It provides a robust framework for evaluating your AI applications and automating the assessment of their performance.

Monday 2025-03-10 Assorted Links

Mon, 10 Mar 2025 00:00:00 +0000

Assorted links for Monday, March 10:

Vector Databases: The Foundation of AI Agent Innovation
NVMe-oF Substantially Reduces Data Access Latency

NVMe-oF is a network protocol that extends the parallel access and low latency features of Nonvolatile Memory Express (NVMe) protocol across networked storage. Originally designed for local storage and common in direct-attached storage (DAS) architectures, NVMe delivers high-speed data access and low latency by directly interfacing with solid-state disks. NVMe-oF allows these same advantages to be achieved in distributed and clustered environments by enabling external storage to perform as if it were local.
Why Observability Needs To Go Headless

Many enterprises generate terabytes of log data every day, resulting in high costs to ingest, store and analyze that data. Even worse, many observability platforms are walled gardens, making it hard to use log data for use cases beyond observability, such as business intelligence, data science and machine learning.

To solve both of these problems, it’s time for headless observability, a fresh approach that decouples the frontend (visualization, querying and analytics) from the backend (data ingestion and storage) — all while keeping operations simple.
The Million-Dollar Problem of Slow Microservices Testing

By shifting integration tests from the slow outer loop into the rapid inner loop, organizations can fundamentally transform their development process.
Strobelight: A profiling service built on open source technology
- We’re sharing details about Strobelight, Meta’s profiling orchestrator.
- Strobelight combines several technologies, many open source, into a single service that helps engineers at Meta improve efficiency and utilization across our fleet.
- Using Strobelight, we’ve seen significant efficiency wins, including one that has resulted in an estimated 15,000 servers' worth of annual capacity savings.

Friday 2025-02-28 Assorted Links

Fri, 28 Feb 2025 00:00:00 +0000

Assorted links for Friday, Febuary 28:

5 Frameworks That Embrace Declarative State Management
The Staging Bottleneck: Microservices Testing in FinTech

A sandbox is a lightweight, isolated, production-like testing setup created dynamically from a shared baseline environment. Designed to replicate production conditions at a fraction of the cost and complexity, sandboxes effectively transform a single staging environment into multiple independent environments. By multiplexing the baseline staging setup, sandboxes provide tailored environments for individual engineers or QA teams without adding compliance risks or increasing maintenance burdens, as they inherit the same compliance and configuration frameworks as production.
Why AI Agents Need an Operational Database
Database Scalability and the Giant Flea: A Lesson in Complexity
How GitHub uses CodeQL to secure GitHub

Thursday 2025-02-27 Assorted Links

Thu, 27 Feb 2025 00:00:00 +0000

Assorted links for Thursday, Febuary 27:

The Engineer’s Guide to Controlling Configuration Drift
Key techniques:
- Infrastructure as Code (IaC)
- Policy as Code (PaC)
- Compliance as Code
- Application Configuration Management
- Configuration Checklist
- Credential Management
- Centralized Configuration Management
- Environment Parity
How Precision Time Protocol handles leap seconds
Observability Isn’t Enough. It’s Time To Federate Log Data

With data federation, you can query data across many different sources without moving it. With this approach, no additional pipeline is needed; there are no egress costs and none of the security risks that come with migrating data.
Data logs: The latest evolution in Meta’s access tools

We created data logs as a solution to provide users who want more granular information with access to data stored in Hive. In this context, an individual data log entry is a formatted version of a single row of data from Hive that has been processed to make the underlying data transparent and easy to understand.
Open Source Redefines Data Platforms

Wednesday 2025-02-26 Assorted Links

Wed, 26 Feb 2025 00:00:00 +0000

Assorted links for Wednesday, Febuary 26:

What is observability 2.0?
Key differences between traditional observability and observability 2.0
- Data handling:
  - Traditional: Relies on separate tools for metrics, logs, and traces, creating silos and requiring manual correlation.
  - 2.0: Unifies telemetry data into a single platform, offering a comprehensive view of system health.
- Problem detection:
  - Traditional Uses static thresholds and alerts that are often reactive and miss subtle issues.
  - 2.0: Employs AI and machine learning to identify anomalies in real-time, enabling proactive issue resolution.
- Focus on context:
  - Traditional: Provides raw technical data without linking it to broader business outcomes.
  - 2.0: Maps telemetry data to business metrics, ensuring decisions align with organizational goals.
- Scalability and adaptability:
  - Traditional Struggles with dynamic environments like Kubernetes and serverless, often requiring custom setups.
  - 2.0: Designed for dynamic scaling, adapts with ease to changes in cloud-native architectures.
Cloud Native Computing Foundation Announces CubeFS Graduation

CubeFS is an open source distributed storage system that supports access protocols such as POSIX, HDFS, S3, and its own REST API. It can be used in many scenarios, including big data, AI/LLMs, container platforms, separation of storage and computing for databases and middleware, data sharing, and more. Key features of CubeFS include a highly scalable metadata service with strong consistency and multi-tenancy support for better resource utilization and tenant isolation.
What Developers Need to Know About Telemetry Pipelines

A telemetry pipeline is a system that collects, processes and routes telemetry data (logs, metrics and traces) from various sources to the right monitoring and analysis tools. Instead of managing separate agents or collectors for different signals, a telemetry pipeline unifies data handling, making observability more efficient and scalable.
What Are Linux Namespaces and How Are They Used?

Namespaces restrict resources that a containerized process can see so that one process can’t see the resources being used by another. This feature is crucial to the likes of containers and orchestration tools such as Kubernetes because, otherwise, one deployed container would be able to access or view resources used by another.
System Operators to Timekeepers: What Will Replace Leap Seconds?

Earth’s rotation, for thousands of years, has mostly slowed, the biggest driver being the changing tides that come with the gravitational tug of the moon. Currents in the planet’s outer core, which scientists are still trying to figure out, also have slowed the spin. But the core can speed up the spin, too, which may be what’s been happening recently. Additional leap seconds have become a lot less frequent in the past two decades.

Tuesday 2025-02-25 Assorted Links

Tue, 25 Feb 2025 00:00:00 +0000

Assorted links for Tuesday, Febuary 25:

Exploring the Inevitable Future of Data-Dependent Applications

Instead of combining technologies like MongoDB, Redis, Kafka, and application servers, why not skip system fragmentation and use a single technology platform? Only code makes these systems run, so why not code for just one system?
Simplify AI Development with Machine Learning Containers
Attacks on Maven proxy repositories

All tested [repository managers] not only store and serve artifacts, but also perform complex parsing and indexing operations on them. Therefore, a specially crafted artifact can be used to attack the repository manager that processes it. This opens a possibility for XSS, XXE, archive expansion, and path traversal attacks.
Modernizing legacy code with GitHub Copilot: Tips and examples
Open source AI is already finding its way into production

In our survey of 2,000 enterprise respondents on software development teams across the US, Germany, India, and Brazil, nearly everyone said they had experimented with open source AI models at some point.

Monday 2025-02-24 Assorted Links

Mon, 24 Feb 2025 00:00:00 +0000

Assorted links for Monday, Febuary 24:

shared_ptr overuse in C++
The pitfall can be divided in two:
- Passing smart pointers to functions that don’t have to deal with ownership
- Improper use of shared_ptr in non-owning objects
.NET 9 Networking Improvements
In this release, we made two impactful performance improvements in HTTP connection pooling.
- We added opt-in support for multiple HTTP/3 connections.
- We also addressed lock contention in HTTP 1.1 connection pooling (dotnet/runtime#70098).
One of the main pain points when debugging HTTP traffic of applications using earlier versions of .NET is that the application doesn’t react to changes in Windows proxy settings. This issue was mitigated in dotnet/runtime#103364, where the HttpClient.DefaultProxy is set to an instance of Windows proxy that listens for registry changes and reloads the proxy settings when notified.
What’s New for ASP.NET Core & Blazor in .NET 9
1. Making Static Files Lightning Fast with MapStaticAssets
2. Blazor Gets Even More Interactive and Hybrid-Friendly
3. Simplified Authentication State Management
Simplify your .NET data transfers with the new Azure Storage Data Movement library
New York Times goes all-in on internal AI tools

The New York Times is greenlighting the use of AI for its product and editorial staff, saying that internal tools could eventually write social copy, SEO headlines, and some code.

Friday 2025-02-21 Assorted Links

Fri, 21 Feb 2025 00:00:00 +0000

Assorted links for Friday, Febuary 21:

How to Store a Knowledge Graph in a Database

A knowledge graph represents information as a set of nodes and the relationships between those nodes.

When your source data consists of assets like technical documentation, research publications, or highly interconnected websites, a knowledge graph returns better results than a simple vector search. That’s because a knowledge graph search can traverse links between nodes, finding semantically relevant results two or more steps away from the first node.
AI Agents Are About to Blow Up the Business Process Layer

Agentic AI is all about autonomy (think self-driving cars), employing a system of agents to constantly adapt to dynamic environments and independently create, execute and optimize results.

When agentic AI is applied to business process workflows, it can replace fragile, static business processes with dynamic, context-aware automation systems.
Storing, querying and keeping embeddings updated: options and best practices
Database and AI: solutions for keeping embeddings updated
- Using a Database Trigger
- Using Change Tracking
- Using an Azure Function Sql Trigger binding
- Using Azure Logic Apps
- Using Change Data Capture
- Using the new Change Event Stream
Introducing Change Event Streaming: Join the Azure SQL Database Private Preview for Change Data Streaming

Thursday 2025-02-20 Assorted Links

Thu, 20 Feb 2025 00:00:00 +0000

Assorted links for Thursday, Febuary 20:

Data Infrastructure, Not AI Models, Will Drive IT Spend in 2025

As organizations race to implement Artificial Intelligence (AI) initiatives, they’re encountering an unexpected bottleneck: the massive cost of data infrastructure required to support AI applications.

I’m seeing organizations address these challenges through innovative architectural approaches. One promising direction is the adoption of leaderless architectures combined with object storage. This approach eliminates the need for expensive data movement by leveraging cloud-native storage solutions that simultaneously serve multiple purposes.

Another key strategy involves rethinking how data is organized and accessed. Rather than maintaining separate infrastructures for streaming and batch processing, companies are moving toward unified platforms that can efficiently handle both workloads. This reduces infrastructure costs and simplifies data governance and access patterns.
Object Store Apps: Cloud Native’s Freshest Architecture

An increasing number of start-ups and end-users find that using cloud object storage as the persistence layer saves money and engineering time that would otherwise be needed to ensure consistency.
The Feds Push for WebAssembly Security Over eBPF

According to a National Institute of Standards and Technology (NIST) paper, “A Data Protection Approach for Cloud-Native Applications” (authors: Wesley Hales from LeakSignal; Ramaswamy Chandramouli, a supervisory computer scientist at NIST), WebAssembly could and should be integrated across the cloud native service mesh sphere in particular to enhance security.
Deep Dive Into DeepSeek-R1: How It Works and What It Can Do

During DeepSeek-R1’s training process, it became clear that by rewarding accurate and coherent answers, nascent model behaviors like self-reflection, self-verification, long-chain reasoning and autonomous problem-solving point to the possibility of emergent reasoning that is learned over time, rather than overtly taught — thus possibly paving the way for further breakthroughs in AI research.
Use Azure Cosmos DB as a Docker container in CI/CD pipelines

The Linux-based Azure Cosmos DB emulator is available as a Docker container and can run on a variety of platforms, including ARM64 architectures like Apple Silicon. It allows local development and testing of applications without needing an Azure subscription or incurring service costs. You can easily run it as a Docker container, and use it for local development and testing.

Wednesday 2025-02-19 Assorted Links

Wed, 19 Feb 2025 00:00:00 +0000

Assorted links for Wednesday, Febuary 19:

AI used to design a multi-step enzyme that can digest some plastics

A new paper today describes a success in making a brand-new enzyme with the potential to digest plastics. But it also shows how even a simple enzyme may have an extremely complex mechanism—and one that’s hard to tackle, even with the latest AI tools.
3 takeaways from red teaming 100 generative AI products
1. Generative AI systems amplify existing security risks and introduce new ones
2. Humans are at the center of improving and securing AI
3. Defense in depth is key for keeping AI systems safe
AIs and Robots Should Sound Robotic

We have a simple proposal: all talking AIs and robots should use a ring modulator. In the mid-twentieth century, before it was easy to create actual robotic-sounding speech synthetically, ring modulators were used to make actors’ voices sound robotic.
2025 OWASP Top 10 for LLM Applications: A Quick Guide
1. LLM01: Prompt injection
2. LLM02: Sensitive information disclosure
3. LLM03: Supply chain
4. LLM04: Data and model poisoning
5. LLM05: Improper output handling
6. LLM06: Excessive agency
7. LLM07: System prompt leakage
8. LLM08: Vector and embedding weaknesses
9. LLM09: Misinformation
10. LLM10: Unbounded consumption
Cloud vs. On-Prem: Which Is Better for Your Kubernetes Cluster?

Cloud solutions offer unparalleled flexibility and ease of scaling, while on-premises setups provide unmatched control and security for sensitive workloads.

Tuesday 2025-02-18 Assorted Links

Tue, 18 Feb 2025 00:00:00 +0000

Assorted links for Tuesday, Febuary 18:

A brief and incomplete comparison of memory corruption detection tools

ASAN detects a lot more types of memory errors, but it requires that you recompile everything. This can be limiting if you suspect that the problem is coming from a component you cannot recompile (say because you aren’t set up to recompile it, or because you don’t have the source code). Valgrind and AppVerifier have the advantage that you can turn them on for a process without requiring a recompilation.
Why Mocks Fail: Real-Environment Testing for Microservices
- Use mocks for edge cases and scenarios requiring controlled inputs.
- Leverage real environments to validate integration flows, complex API behaviors and performance characteristics against real dependencies.
Emerging Patterns in Building GenAI Products:
- Direct Prompting: Send prompts directly from the user to a Foundation LLM
- Embeddings: Transform large data blocks into numeric vectors so that embeddings near each other represent related concepts
- Evals: Evaluate the responses of an LLM in the context of a specific task
- Hybrid Retriever: Combine searches using embeddings with other search techniques
- Query Rewriting: Use an LLM to create several alternative formulations of a query and search with all the alternatives
- Reranker: Rank a set of retrieved document fragments according to their usefulness and send the best of them to the LLM.
- Retrieval Augmented Generation (RAG): Retrieve relevant document fragments and include these when prompting the LLM
How Meta discovers data flows via lineage at scale

In order to build high-quality data lineage, we developed different techniques to collect data flow signals across different technology stacks: static code analysis for different languages, runtime instrumentation, and input and output data matching, etc.
Sam Altman lays out roadmap for OpenAI’s long-awaited GPT-5 model

GPT-5 will be a system that brings together features from across OpenAI’s current AI model lineup, including conventional AI models, SR models, and specialized models that do tasks like web search and research.

Monday 2025-02-17 Assorted Links

Mon, 17 Feb 2025 00:00:00 +0000

Assorted links for Monday, Febuary 17:

Time Bandit ChatGPT jailbreak bypasses safeguards on sensitive topics

A ChatGPT jailbreak flaw, dubbed “Time Bandit,” allows you to bypass OpenAI’s safety guidelines when asking for detailed instructions on sensitive topics, including the creation of weapons, information on nuclear topics, and malware creation.
A US Treasury Threat Intelligence Analysis Designates DOGE Staff as ‘Insider Threat’

An internal email reviewed by WIRED calls DOGE staff’s access to federal payments systems “the single greatest insider threat risk the Bureau of the Fiscal Service has ever faced.”
Optimizing for Developer Productivity Creates a Winning DevEx

Developer productivity is not about having 50 tools. It’s about improving experience, speed and productivity with the right kinds of tools.
Microsoft.Testing.Platform: Now Supported by All Major .NET Test Frameworks

Microsoft.Testing.Platform is a lightweight and portable alternative to VSTest for running tests in all contexts, including continuous integration (CI) pipelines, CLI, Visual Studio Test Explorer, and VS Code Text Explorer. The Microsoft.Testing.Platform is embedded directly in your test projects, and there’s no other app dependencies, such as vstest.console or dotnet test needed to run your tests.
OpenAI’s secret weapon against Nvidia dependence takes shape

OpenAI is entering the final stages of designing its long-rumored AI processor with the aim of decreasing the company’s dependence on Nvidia hardware, according to a Reuters report released Monday. The ChatGPT creator plans to send its chip designs to Taiwan Semiconductor Manufacturing Co. (TSMC) for fabrication within the next few months, but the chip has not yet been formally announced.

Tuesday 2025-01-21 Assorted Links

Tue, 21 Jan 2025 00:00:00 +0000

Assorted links for Tuesday, January 21:

Monday 2025-01-20 Assorted Links

Mon, 20 Jan 2025 00:00:00 +0000

Assorted links for Monday, January 20:

The Boring Option: Migrating Segment Efforts Storage at Strava
How do you test your tests?
How Facebook keeps its large-scale infrastructure hardware up and running
JUring: Experimental IO_uring For Java With Big Performance Gains

JUring is a high-performance Java library that provides bindings to Linux’s io_uring asynchronous I/O interface using Java’s Foreign Function & Memory API. Doing Random reads JUring achieves 33% better performance than Java NIO FileChannel operations for local files and 78% better performance for remote files.
How we built the GitHub Skyline CLI extension using GitHub

Friday 2025-01-17 Assorted Links

Fri, 17 Jan 2025 00:00:00 +0000

Assorted links for Friday, January 17:

Thursday 2025-01-16 Assorted Links

Thu, 16 Jan 2025 00:00:00 +0000

Assorted links for Thursday, January 16:

Fast commits for ext4

The Linux 5.10 release included a change that is expected to significantly increase the performance of the ext4 filesystem; it goes by the name “fast commits” and introduces a new, lighter-weight journaling method.
Building Faster AMD64 Memset Routines
How NuGet resolves package dependencies
Maximizing Developer Effectiveness
What’s good about offset pagination; designing parallel cursor-based web APIs

Wednesday 2025-01-15 Assorted Links

Wed, 15 Jan 2025 00:00:00 +0000

Assorted links for Wednesday, January 15:

Cloud PUE: Comparing AWS, Azure and GCP Global Regions

New data reveals how efficiently the major cloud providers run and cool their data centers – from AWS’s and Azure’s tropical struggles to Google’s industry-leading performance.
The No-Order File System

In this paper, we introduce the No-Order File System (NoFS), a simple, lightweight file system that employs a novel technique called backpointer based consistency to provide crash consistency without ordering writes as they go to disk.
Whose Code is it Anyway?

In order to measure the engineering effectiveness of Yelp, we need to measure the effectiveness of its organizations and the teams that make up those organizations. But how do we know what a team is responsible for? We needed a way to assign an owner to something (let’s call this an entity) that we want to measure. Once an entity has an owner, we can collect metrics on that entity and derive the health score (i.e., effectiveness) for that owner. These metrics can then be aggregated by team, organization, or even the entire Engineering division, so that we can identify areas that we can collectively improve. And this is how the Ownership microservice was born.
How we ported Linux to the M1
Nix + Bazel = fully reproducible, incremental builds

Tuesday 2025-01-14 Assorted Links

Tue, 14 Jan 2025 00:00:00 +0000

Assorted links for Tuesday, January 14:

αcτµαlly pδrταblε εxεcµταblε

One day, while studying old code, I found out that it’s possible to encode Windows Portable Executable files as a UNIX Sixth Edition shell script, due to the fact that the Thompson Shell didn’t use a shebang line. Once I realized it’s possible to create a synthesis of the binary formats being used by Unix, Windows, and MacOS, I couldn’t resist the temptation of making it a reality, since it means that high-performance native code can be almost as pain-free as web apps.
I spent 4 hours learning how Netflix operates Apache Iceberg at scale
How to secure your GitHub Actions workflows with CodeQL

To help prevent the introduction of vulnerabilities, identify them in existing workflows, and even fix them using GitHub Copilot Autofix, CodeQL support has been added for GitHub Actions.
How the UK was connected to the Internet for the first time
Highlights from Git 2.48

The open source Git project just released Git 2.48. Here is GitHub’s look at some of the most interesting features and changes introduced since last time.

Monday 2025-01-13 Assorted Links

Mon, 13 Jan 2025 00:00:00 +0000

Assorted links for Monday, January 13:

Simple defer, ready to use

With this post I will concentrate on the here and now: how to use C’s future lifesaving defer feature with existing tools and compilers.
Cleanup Attribute in C

In this blog post I explore __attribute__((cleanup(...))). I discuss what it does, how it does it, why use it, performance considerations, and finish by saying it’s absolutely fantastic.
Debanking (and Debunking?): An entertaining and fairly deep explanation of what debanking is and why it occurs. And I learned a useful new word: sontaku.

Japanese has a beautiful word, sontaku, for the attitude and actions a diligent subordinate would take without his superior’s explicit instruction, believing them to anticipate his boss’ desires. Sontaku is a core skill in the American professional class.
Seeing like a Bank
The Bond villain compliance strategy

Friday 2025-01-10 Assorted Links

Fri, 10 Jan 2025 00:00:00 +0000

Assorted links for Friday, January 10:

Colliding with the SHA prefix of Linux’s initial Git commit

There was a recent discussion about how Linux’s “Fixes” tag, which traditionally uses the 12 character commit SHA prefix, has an ever increasing chance of collisions. There are already 11-character collisions, and Geert wanted to raise the minimum short id to 16 characters.

Tools like linux-next’s “Fixes tag checker”, the Linux CNA’s commit parser, and my own CVE lifetime analysis scripts do programmatic analysis of the “Fixes” tag and had no support for collisions (even shorter existing collisions).

So, in an effort to fix these tools, I broke them with commit 1da177e4c3f4 (“docs: git SHA prefixes are for humans”):
FreeBSD Considers Making Use Of Rust Within Its Base System
The Architect’s Guide to Open Table Formats and Object Storage
Alerts Are Fundamentally Messy
Databases in 2024: A Year in Review

Thursday 2025-01-09 Assorted Links

Thu, 09 Jan 2025 00:00:00 +0000

Assorted links for Thursday, January 9:

Wednesday 2025-01-08 Assorted Links

Wed, 08 Jan 2025 00:00:00 +0000

Assorted links for Wednesday, January 8:

Amex’s FaaS Uses WebAssembly Instead of Containers

A key reason behind Amex’s adoption of WebAssembly is that WebAssembly demonstrated superior performance metrics compared to containers.
Enhance build security and reach SLSA Level 3 with GitHub Artifact Attestations

The Supply-chain Levels for Software Artifacts (SLSA) framework … provides a comprehensive, step-by-step methodology for building integrity and provenance guarantees into your software supply chain.
Introducing Configurable Metaflow

Standing on the shoulders of our extensive cloud infrastructure, Metaflow facilitates easy access to data, compute, and production-grade workflow orchestration, as well as built-in best practices for common concerns such as collaboration, versioning, dependency management, and observability, which teams use to setup ML/AI experiments and systems that work for them. As a result, Metaflow users at Netflix have been able to run millions of experiments over the past few years without wasting time on low-level concerns.
The Feds Push WebAssembly for Cloud Native Security

According to a National Institute of Standards and Technology (NIST) paper, “A Data Protection Approach for Cloud-Native Applications,” released earlier this year, WebAssembly could and should be integrated across the cloud native service mesh sphere in particular to enhance security.
Self-Designing Software

Exploring ways to include a software system as an active member of its own design team, able to reason about its own design and to synthesize better variants of its own building blocks as it encounters different deployment conditions.

Tuesday 2025-01-07 Assorted Links

Tue, 07 Jan 2025 00:00:00 +0000

Assorted links for Tuesday, January 7:

What is Inference Parallelism and how it works

Inference parallelism aims to distribute the computational workload of AI models, particularly deep learning models, across multiple processing units such as GPUs.
Open Source Innovation Comes to Time-Series Data Compression

NetApp Instaclustr collaborated with the University of Canberra through the OpenSI initiative to develop the Advanced Time Series Compressor (ATSC) — an open source innovation that fundamentally reimagines high-volume time-series data compression.

ATSC implements a sophisticated lossy compression approach. Rather than storing complete data sets, it generates mathematical functions that closely approximate the original data patterns, storing only the essential parameters of these functions. This approach is paired with granular configurability — users can precisely tune their desired level of accuracy, balancing storage efficiency with data fidelity based on their specific use cases.
What Do You Lose When You Abandon the Cloud?

High-profile moves from 37signals (the company behind Basecamp and HEY) and GEICO have sparked a renewed interest in cloud repatriation.

One sometimes overlooked advantage of moving to the cloud is that it allows you to pay for resources when they are needed, for example, as new customers come online. Spending moves from upfront CAPEX (buying new machines in anticipation of success) to OPEX (paying for additional servers on demand).

Another thing to weigh up is pace of innovation — both from the cloud provider and from the consumer.

The Zynga example [of moving from the cloud to on-prem, then back to the cloud] highlights several other trade-offs. One to consider is that if you are running your own data centers, you need to be able to hire the right people and retain them.

There is another set of trade-offs around security. Keeping servers up to date, and guarding against intrusions, is time-consuming work that big cloud providers are very experienced in.
Why All the Major Cloud Platforms Are the Same

Each provider brought unique strengths and strategic priorities to the table, creating differentiation initially, but eventually converging on a consistent baseline of functionality.
Indexing code at scale with Glean
How is Glean different?
- Glean doesn’t decide for you what data you can store.
- Glean’s query language is very general.

Monday 2025-01-06 Assorted Links

Mon, 06 Jan 2025 00:00:00 +0000

Assorted links for Monday, January 6:

Managing large-scale Redis clusters on Kubernetes with an operator – Kuaishou’s approach
Supercharge Your RAG App With Agentic Hybrid Search

By using structured metadata and letting an LLM choose the best retrieval method for each query, you can turn your RAG app into a better assistant.
Cloud Efficiency at Netflix
Part 1: A Survey of Analytics Engineering Work at Netflix
How we think about Threads’ iOS performance: Key metrics: %FIRE (Frustrating image-render experience), TTNC (Time-to-network content), cPSR (Creation-publish success rate)

Friday 2025-01-03 Assorted Links

Fri, 03 Jan 2025 00:00:00 +0000

Assorted links for Friday, January 3:

Apache Pinot Brings Real Time Analysis to Columnar Data

Pinot was born to solve the problem of “running analytical queries for hundreds of millions of users at scale, in a low-cost manner,” explained Chinmay Soman, head of product for StarTree, which offers a fully managed cloud native version of Pinot.

Pinot brings “simplification in the data stack,” Soman said in an interview with TNS. “The problem is not new. It’s been solved by many legacy technologies. What Pinot brings is the simplification and the scale for these problems.”
GenAI is Quickly Reinventing IT Operations, Leaving Many Behind

GenAI can significantly transform IT Operations Management by proactively providing context-rich insights, accurate predictions, and actionable recommendations for managing the IT landscape.
How to Create and Use an AI Git Agent
Platform Engineering needs Observability: here’s why

Observability offers real-time insights into system behavior, allowing teams to proactively identify and address issues before they affect users. By adopting observability, platform engineering teams can improve system resilience, sustain uninterrupted user experiences during peak events, and uphold operational stability.
Announcing systemd v257

Thursday 2025-01-02 Assorted Links

Thu, 02 Jan 2025 00:00:00 +0000

Assorted links for Thursday, January 2:

Design Token-Based UI Architecture

Design tokens are design decisions as data and serve as a single source of truth for design and engineering. Utilizing deployment pipelines, they enable automated code generation across platforms, allowing for faster updates and improved consistency in design.
Retrofitting spatial safety to hundreds of millions of lines of C++

[W]e’re working towards bringing spatial memory safety into as many of our C++ codebases as possible, including Chrome and the monolithic codebase powering our services.

…

Building on the successful deployment of hardened libc++ in Chrome in 2022, we’ve now made it default across our server-side production systems.
Reconsidering Kubernetes deployments: when operators are overkill

TL:DR: Kubernetes Operators are powerful but can be overkill for simple deployments. Explore alternatives like Helm, ArgoCD, and Devtron to streamline your Kubernetes deployments without sacrificing efficiency.
Securely Deploy and Run Multiple Tenants on Kubernetes
Three fundamental options:
1. Namespace-Based Isolation With Network Policies, RBAC and Security Controls
2. Cluster-Level Isolation
3. Virtual Clusters
AI-Driven Code Review: Enhancing Developer Productivity and Code Quality

AI-driven code review tools are changing traditional development and helping teams build better software.

Tuesday 2024-12-10 Assorted Links

Tue, 10 Dec 2024 00:00:00 +0000

Assorted links for Tuesday, December 10:

Benchmark LLM Application Performance with LangChain
Forget All-Cloud or All-On-Prem: Embrace Hybrid for Agility and Cost Savings

The core idea behind hybrid cloud is straightforward: use cloud environments where rapid scaling and frequent changes are essential and rely on on-premises systems for stable, predictable workloads.
Bazel 8.0 Released
You Can Now Try Out Sora, OpenAI’s AI Video Generator
Using Local AI models with .NET Aspire

Monday 2024-12-09 Assorted Links

Mon, 09 Dec 2024 00:00:00 +0000

Assorted links for Monday, December 9:

Friday 2024-12-06 Assorted Links

Fri, 06 Dec 2024 00:00:00 +0000

Assorted links for Friday, December 6:

Thursday 2024-12-05 Assorted Links

Thu, 05 Dec 2024 00:00:00 +0000

Assorted links for Thursday, December 5:

Wednesday 2024-12-04 Assorted Links

Wed, 04 Dec 2024 00:00:00 +0000

Assorted links for Wednesday, December 4:

Tuesday 2024-12-03 Assorted Links

Tue, 03 Dec 2024 00:00:00 +0000

Assorted links for Tuesday, December 3:

Agent Framework

Agent Framework is a Python framework for building capable, context-aware AI agents. It provides a flexible foundation for creating AI agents with distinct personas, managed instructions, conversational memory, and strategic planning capabilities.
eBPF Security Threat Model
Experts Share Best Practices for Building Terraform Modules
API Mocking Is Essential to Effective Change Management
OpenTelemetry Is expanding into CI/CD observability

We’ve been talking about the need for a common “language” for reporting and observing CI/CD pipelines for years, and finally, we see the first “words” of this language entering the “dictionary” of observability – the OpenTelemetry open specification. With the recent release of OpenTelemetry’s Semantic Conventions, v1.27.0, you can find designated attributes for reporting CI/CD pipelines.

Monday 2024-12-02 Assorted Links

Mon, 02 Dec 2024 00:00:00 +0000

Assorted links for Monday, December 2:

Cloud Neutral Postgres Databases with Kubernetes and CloudNativePG
Observe Simplifies K8s Troubleshooting With Kubernetes Explorer
Platform Engineering: A Workshop to Help Map Your Strategy
Revisiting the DOS memory models
Linux Kernel Performance Bottlenecks Spotted By Mold Developer
The key takeaways of the performance bottlenecks within the Linux kernel that were noticed by Rui Ueyama include:
- exit(2) takes a few hundred milliseconds for a large process
- Writing to a fresh file is slower than writing to an existing file
- Lack of a safe system-wide semaphore

Friday 2024-11-29 Assorted Links

Fri, 29 Nov 2024 00:00:00 +0000

Assorted links for Friday, November 29:

The Case for a High-level Kernel-Bypass I/O Abstraction: Related to Demikernel
Chaos Under Control: Addressing Cloud Infrastructure Drift
Ignite 2024: Microsoft Debuts SQL Server 2025, Integrates Azure SQL into Fabric
BazelCon 2024 Recap: Recordings and Birds of a Feather Session Notes
Emerging trends in the cloud native ecosystem
1. Cloud Cost Control Gets Cool: FinOps is the New Black
2. Are Your Devs Drowning in Tools? IDPs Are the Life Raft!
3. Beyond Silos: Open Standards Unify Observability
4. Cloud-native Security: A New High-stakes Game
5. The Future is Green: Why Sustainable IT is No Longer Optional

Thursday 2024-11-28 Assorted Links

Thu, 28 Nov 2024 00:00:00 +0000

Assorted links for Thursday, November 28:

What is Bicep?: Looks like an Azure-only Terraform.

Bicep is a domain-specific language that uses declarative syntax to deploy Azure resources. In a Bicep file, you define the infrastructure you want to deploy to Azure and then use that file throughout the development lifecycle to repeatedly deploy that infrastructure. Your resources are deployed in a consistent manner.
Netflix’s Distributed Counter Abstraction
Hundreds of code libraries posted to NPM try to install malware on dev machines: Make sure you SHA-lock your dependencies!
From String to AST: parsing
Quake 3 Source Code Review

Wednesday 2024-11-27 Assorted Links

Wed, 27 Nov 2024 00:00:00 +0000

Assorted links for Wednesday, November 27:

Tuesday 2024-11-26 Assorted Links

Tue, 26 Nov 2024 00:00:00 +0000

Assorted links for Tuesday, November 26:

What does f(x) mean in C++?: A vivid example of just how complex of a language C++ is.
Scaling From Simple to Complex Cache: Challenges and Solutions
An Engineer’s Checklist of Logging Best Practices
Kloudfuse 3.0: an All-in-One Observability Platform Emerges
Why Testcontainers Cloud Is a Game-Changer Compared to Docker-in-Docker for Testing Scenarios

Monday 2024-11-25 Assorted Links

Mon, 25 Nov 2024 00:00:00 +0000

Assorted links for Monday, November 25:

Introducing the .NET Aspire Community Toolkit

The team behind the Community Toolkit wanted to create a place where third parties could contribute their own integrations and extensions to the .NET Aspire ecosystem.
Calling methods is easier and faster with C# 13 params collections

C# 13 fulfills a long standing feature request by allowing params to be any of the collections supported by collection expressions, rather than just arrays.
Announcing .NET 9
Graph RAG: How To Squeeze More Value From AI
How to Support Streaming in AI Applications with LangChain and Langflow

Friday 2024-11-22 Assorted Links

Fri, 22 Nov 2024 00:00:00 +0000

Assorted links for Friday, November 22:

Kernel optimization with BOLT

Generally, developers think about how the data structures for their programs will be arranged in memory; it is less common for them to consider how the code is arranged. There are exceptions, including the Linux kernel developers, but most times the focus is on the data cache. The instruction cache is much smaller than the data cache and has not grown much over time, maybe doubling from 32KB to 64KB on Intel CPUs over the last 20 years. But, for large applications that do not spend most of their time in tight loops, the layout of code in memory matters a lot, so BOLT can make a major difference, he said.
Cache vs. Database: Has Performance Converged?

Both memcached and ScyllaDB maximized disks and network bandwidth while being stressed under similar conditions, sustaining similar performance overall.
Deterministic Simulation Testing
How We Built a Self-Healing System to Survive a Terrifying Concurrency Bug At Netflix
Optimizers: The Low-Key MVP

The query optimizer is an important part of any analytical database system as it provides considerable performance improvements compared to hand-optimized queries, even as the state of your data changes.

Thursday 2024-11-21 Assorted Links

Thu, 21 Nov 2024 00:00:00 +0000

Assorted links for Thursday, November 21:

Why You No Longer Need Cloud Security Posture Management
Bringing developer choice to Copilot with Anthropic’s Claude 3.5 Sonnet, Google’s Gemini 1.5 Pro, and OpenAI’s o1-preview: It seems the general trend is towards multi-model choice.
Kubectl logs tail: a guide to tailing Kubernetes logs
AI Testing: More Coverage, Fewer Bugs, New Risks
Proactive Monitoring Will Maximize Your Cloud Storage Efficiency

Wednesday 2024-11-20 Assorted Links

Wed, 20 Nov 2024 00:00:00 +0000

Assorted links for Wednesday, November 20:

Tuesday 2024-11-19 Assorted Links

Tue, 19 Nov 2024 00:00:00 +0000

Assorted links for Tuesday, November 19:

Why doesn’t Cloudflare use containers in their Workers platform infrastructure?
P99Conf: How eBPF Could Make Faster Database Systems: Basically this meanings running the database in kernel mode as an eBPF module. At that point, why not use a unikernel?
2024 DORA Report
4 Ways Organizations Can Simplify Their DevOps Pipeline
Don’t Trust Security in AI-Generated Code

Monday 2024-11-18 Assorted Links

Mon, 18 Nov 2024 00:00:00 +0000

Assorted links for Monday, November 18:

Friday 2024-11-15 Assorted Links

Fri, 15 Nov 2024 00:00:00 +0000

Assorted links for Friday, November 15:

How To Create Software Diagrams With ChatGPT and Claude
VMware Workstation Shifting From Proprietary Code To Using Upstream KVM: I see a lot of “as open source evolves, replace proprietary internal code with open source code”-type projects at work. This probably should be written up as an expected, natural evolution of a software system.
Announcing EAP for Vector Support in Azure SQL Database
You Can Now Search the Internet With ChatGPT
OpenCost: Open source cost monitoring for cloud native environments: Cost monitoring for Kubernetes

Thursday 2024-11-14 Assorted Links

Thu, 14 Nov 2024 00:00:00 +0000

Assorted links for Thursday, November 14:

Wednesday 2024-11-13 Assorted Links

Wed, 13 Nov 2024 00:00:00 +0000

Assorted links for Wednesday, November 13:

CMake Implementation of OpenSSF Compiler Hardening Options

Tue, 12 Nov 2024 00:00:00 +0000

This post describes how to implement the OpenSSF Compiler Options Hardening Guide for C and C++ in CMake.

Tuesday 2024-11-12 Assorted Links

Tue, 12 Nov 2024 00:00:00 +0000

Assorted links for Tuesday, November 12:

Monday 2024-11-11 Assorted Links

Mon, 11 Nov 2024 00:00:00 +0000

Assorted links for Monday, November 11:

Move Semantics for IDisposable Part 2

Mon, 11 Nov 2024 00:00:00 +0000

After trying out the Movable<TResource> type from Move Semantics for IDisposable I discovered a fatal flaw in its implementation: it is incompatible with struct memberwise copy semantics.

Friday 2024-11-08 Assorted Links

Fri, 08 Nov 2024 00:00:00 +0000

Assorted links for Friday, November 8:

Uber’s Real-Time Push Platform
Data Mesh Principles and Logical Architecture
How to unit-test your helm charts with Golang
RClone: A command-line program to sync files and directories to and from different cloud storage providers
Different I/O Access Methods for Linux, What We Chose for ScyllaDB, and Why: This post predates io_uring which is probably the modern preference.

Move Semantics for IDisposable

Fri, 08 Nov 2024 00:00:00 +0000

C++ 11 introduced the concept of move semantics to model transfer of ownership. Rust includes transfer of ownership as a key component of its type system. C# could benefit from something similar for IDisposable types. This blog post explores some options on how to handle this.

Thursday 2024-11-07 Assorted Links

Thu, 07 Nov 2024 00:00:00 +0000

Assorted links for Thursday, November 7:

Wednesday 2024-11-06 Assorted Links

Wed, 06 Nov 2024 00:00:00 +0000

Assorted links for Wednesday, November 6:

Tuesday 2024-11-05 Assorted Links

Tue, 05 Nov 2024 00:00:00 +0000

Assorted links for Tuesday, November 5:

Monday 2024-11-04 Assorted Links

Mon, 04 Nov 2024 00:00:00 +0000

Assorted links for Monday, November 4:

Friday 2024-11-01 Assorted Links

Fri, 01 Nov 2024 00:00:00 +0000

Assorted links for Friday, November 1:

AI Flame Graphs
OpenZFS deduplication is good now and you shouldn’t use it
Apache OpenDAL: Open Data Access Layer: Access data freely
Embeddings are underrated

Embeddings aren’t exactly new, but they have become much more widely accessible in the last couple years. What embeddings offer to technical writers is the ability to discover connections between texts at previously impossible scales.
How To Fail at Microservices

Thursday 2024-10-31 Assorted Links

Thu, 31 Oct 2024 00:00:00 +0000

Assorted links for Thursday, October 31:

The Future of APIs: Lessons in Security, Composability, AI
How Apollo Makes LLMs More Reliable with GraphQL

GraphQL’s structured query language proves to be an ideal match for AI tools, offering more reliable and focused results compared to broader LLM applications.
How useful is the hint passed to the std::unordered_… collections?: An unsorted followup on Speeding up the insertion of a sorted (or mostly-sorted) key list into a std::map or other ordered associative container
What is a Vector Database & How Does it Work? Use Cases + Examples
What is Retrieval-Augmented Generation (RAG)?

Wednesday 2024-10-30 Assorted Links

Wed, 30 Oct 2024 00:00:00 +0000

Assorted links for Wednesday, October 30:

Finding the Right Data Architecture for RAG Pipelines

Event streaming is ideal for RAG pipelines that feed generative AI applications the contextual data they need to produce accurate, timely results.
How to Run Databases on Kubernetes: An 8-Step Guide
AI Code Assistants Are Moving Beyond Auto-Complete: Here’s What’s Next
Rustls Outperforms OpenSSL and BoringSSL: Is there something intrinsic to Rust, or is this an artifact of more time spent on optimization?
What is zero trust authorization?: This doesn’t seem novel to me – it seems obvious and is how I’ve architected software for 15+ years.

Never trust, always verify: The ZTA model presumes that no entity—be it a user, device, or software—warrants automatic confidence, irrespective of their physical position or historical clearance levels. Each request for entry must undergo scrutiny.

Tuesday 2024-10-29 Assorted Links

Tue, 29 Oct 2024 00:00:00 +0000

Assorted links for Tuesday, October 29:

Monday 2024-10-28 Assorted Links

Mon, 28 Oct 2024 00:00:00 +0000

Assorted links for Monday, October 28:

IPLS: Privacy-preserving storage for your WhatsApp contacts
New discovery reveals how diatoms capture CO₂ so effectively
What Color is Your Function?
Angular’s Approach to Partial Hydration

Rather than fully hydrate the application immediately, partial hydration allows developers to identify portions of their application — maybe a footer or something that a user will not immediately need to see — and rather than ship all of the JavaScript in the app, it “hydrates” only the parts that are needed immediately.
AWS Welcomes the OpenSearch Software Foundation

OpenSearch, the popular open source, Apache 2.0-licensed, search and analytics suite, is celebrating a significant milestone - transferring OpenSearch to the OpenSearch Software Foundation, a community-driven initiative under the Linux Foundation.

Friday 2024-10-25 Assorted Links

Fri, 25 Oct 2024 00:00:00 +0000

Assorted links for Friday, October 25:

How we build GitHub Copilot into Visual Studio
Sampling with SQL
Basecamp-maker 37Signals says its “cloud exit” will save it $10M over 5 years: The cloud is many things, but “cheap” is not one of them.
Federal Trade Commission Announces Final Rule Banning Fake Reviews and Testimonials
Debunking Hype: China Hasn’t Broken Military Encryption With Quantum

Thursday 2024-10-24 Assorted Links

Thu, 24 Oct 2024 00:00:00 +0000

Assorted links for Thursday, October 24:

Safer with Google: Advancing Memory Safety
Announcing Rust 1.82.0
AWS Makes ETL Disappear for Aurora PostgreSQL, DynamoDB

This week, Amazon Web Services introduced new integrations with its Amazon Aurora PostgreSQL and Amazon DynamoDB database management services that allow them to share data with the Amazon Redshift data warehouse services, without the need to set up ETL (Extract, Transform and Load) workflows between them.
Federal Trade Commission Announces Final “Click-to-Cancel” Rule Making It Easier for Consumers to End Recurring Subscriptions and Memberships: Since Loper Bright Enterprises v. Raimondo overturned the Chevron doctrine, I expect this rule to be challenged immediately. Update 2024-10-29: Comcast, Charter Sue FTC Over Efforts To Make Canceling Services Easier
The Architect’s Guide to Interoperability in the AI Data Stack

Wednesday 2024-10-23 Assorted Links

Wed, 23 Oct 2024 00:00:00 +0000

Assorted links for Wednesday, October 23:

Startup can identify deepfake video in real time
Announcing IBM Granite AI Models Now Available on Docker Hub
Graph RAG: How To Squeeze More Value From AI
Millions of Vehicles Could Be Hacked and Tracked Thanks to a Simple Website Bug

Researchers found a flaw in a Kia web portal that let them track millions of cars, unlock doors, and start engines at will—the latest in a plague of web bugs that’s affected a dozen carmakers.
OpenHCL: the new, open source paravisor

A paravisor executes within the confidential trust boundary and provides the virtualization and device services needed by a general-purpose operating system (OS), enabling existing VM workloads to execute securely without requiring continual service of the OS to take advantage of innovative advances in confidential computing technology. As confidential computing becomes available on more hardware platforms and evolves, the software stack can keep VMs running seamlessly thanks to the paravisor, in much the same way other advances in virtualization software enabled VMs to run seamlessly on ever evolving hardware.

Tuesday 2024-10-22 Assorted Links

Tue, 22 Oct 2024 00:00:00 +0000

Assorted links for Tuesday, October 22:

What’s new in System.Text.Json in .NET 9

The 9.0 release of System.Text.Json includes many features, primarily with a focus on JSON schema and intelligent application support. It also includes highly requested enhancements such as nullable reference type support, customizing enum member names, out-of-order metadata deserialization and customizing serialization indentation.
A quick introduction to return address protection technologies
North Korean hackers use newly discovered Linux malware to raid ATMs

The malware, tracked under the name FASTCash, is a remote access tool that gets installed on payment switches inside compromised networks that handle payment card transactions.

The purpose of FASTCash is to compromise a key switch inside the complex networks that broker payment transactions among merchants and their banks on the one hand and, on the other, the payment card issuers who must approve a transaction… When a compromised card is used to make a fraudulent translation, FASTCash tampers with the messages the switch receives from issuers before relaying it back to the merchant bank. As a result, issuer messages denying the transaction are changed to approvals.
Building Knowledge Graphs at Production Scale for GenAI
Docker Best Practices: Using ARG and ENV in Your Dockerfiles

If you need to access a variable during the build process but not at runtime, use ARG. If you need to access the variable both during the build and at runtime, or only at runtime, use ENV.

Monday 2024-10-21 Assorted Links

Mon, 21 Oct 2024 00:00:00 +0000

Assorted links for Monday, October 21:

Thousands of Linux systems infected by stealthy malware since 2021
Invisible text that AI chatbots understand and humans can’t? Yep, it’s a thing.

The invisible characters, the result of a quirk in the Unicode text encoding standard, create an ideal covert channel that can make it easier for attackers to conceal malicious payloads fed into an LLM. The hidden text can similarly obfuscate the exfiltration of passwords, financial information, or other secrets out of the same AI-powered bots. Because the hidden text can be combined with normal text, users can unwittingly paste it into prompts.
Investigation of a Workbench UI Latency Issue: Resolving a UI latency investigation requires from the UI all the way down to the Linux kernel.
Web Browser Engineering by Pavel Panchekha & Chris Harrelson
Drasi: Drasi is a data processing platform that simplifies detecting changes in data and taking immediate action. It is a comprehensive solution that provides built-in capabilities to track system logs and change feeds for specific events, evaluate them for relevance, and automatically initiate appropriate reactions.

Thursday 2024-10-17 Assorted Links

Thu, 17 Oct 2024 00:00:00 +0000

Assorted links for Thursday, October 17:

China Possibly Hacking US “Lawful Access” Backdoor

The Wall Street Journal is reporting that Chinese hackers (Salt Typhoon) penetrated the networks of US broadband providers, and might have accessed the backdoors that the federal government uses to execute court-authorized wiretap requests. Those backdoors have been mandated by law—CALEA—since 1994.
The second half of software supply chain security on GitHub
For the first time since 1882, UK will have no coal-fired power plants
Introducing Netflix’s TimeSeries Data Abstraction Layer
Two never-before-seen tools, from same group, infect air-gapped devices

Thursday 2024-10-17 Assorted Links

Thu, 17 Oct 2024 00:00:00 +0000

Assorted links for Thursday, October 17:

China Possibly Hacking US “Lawful Access” Backdoor

The Wall Street Journal is reporting that Chinese hackers (Salt Typhoon) penetrated the networks of US broadband providers, and might have accessed the backdoors that the federal government uses to execute court-authorized wiretap requests. Those backdoors have been mandated by law—CALEA—since 1994.
The second half of software supply chain security on GitHub
For the first time since 1882, UK will have no coal-fired power plants
Introducing Netflix’s TimeSeries Data Abstraction Layer
Two never-before-seen tools, from same group, infect air-gapped devices

Wednesday 2024-10-16 Assorted Links

Wed, 16 Oct 2024 00:00:00 +0000

Assorted links for Wednesday, October 16:

Tuesday 2024-10-15 Assorted Links

Tue, 15 Oct 2024 00:00:00 +0000

Assorted links for Tuesday, October 15:

Artist appeals copyright denial for prize-winning AI-generated work
Neighbors sue over loud Bitcoin mine

Hum from 300-megawatt facility allegedly causing stress, lack of sleep.
Highlights from Git 2.47
What’s New In Python 3.13
Reports: China hacked Verizon and AT&T, may have accessed US wiretap systems

Monday 2024-10-14 Assorted Links

Mon, 14 Oct 2024 00:00:00 +0000

Assorted links for Monday, October 14:

Attackers exploit critical Zimbra vulnerability using cc’d email addresses
Leveraging Graph Databases for Fraud Detection in Financial Systems

Graph databases reveal patterns and relationships that would otherwise be hidden, allowing financial institutions to detect fraud faster and more efficiently.
Microsoft’s new “Copilot Vision” AI experiment can see what you browse
The more sophisticated AI models get, the more likely they are to lie
How Cloudflare auto-mitigated world record 3.8 Tbps DDoS attack

Wednesday 2024-10-02 Assorted Links

Wed, 02 Oct 2024 00:00:00 +0000

Assorted links for Wednesday, October 2:

Tuesday 2024-10-01 Assorted Links

Tue, 01 Oct 2024 00:00:00 +0000

Assorted links for Tuesday, October 1:

Hacker plants false memories in ChatGPT to steal user data in perpetuity
Getting started with testing and .NET Aspire
The 10 best tools to green your software
Introducing NotebookLM

An AI-first notebook, grounded in your own documents, designed to help you gain insights faster.
NotebookLM now lets you listen to a conversation about your sources

Our new Audio Overview feature can turn documents, slides, charts and more into engaging discussions with one click.

Monday 2024-09-30 Assorted Links

Mon, 30 Sep 2024 00:00:00 +0000

Assorted links for Monday, September 30:

Changes coming in PostgreSQL 17
Eliminating Memory Safety Vulnerabilities at the Source
Unauthenticated RCE Flaw With CVSS 9.9 Rating For Linux Systems Affects CUPS

A remote unauthenticated attacker can silently replace existing printers' (or install new ones) IPP urls with a malicious one, resulting in arbitrary command execution (on the computer) when a print job is started (from that computer)."
NIST Recommends Some Common-Sense Password Rules
Microsoft details security/privacy overhaul for Windows Recall ahead of relaunch

Tuesday 2024-09-24 Assorted Links

Tue, 24 Sep 2024 00:00:00 +0000

Assorted links for Tuesday, September 24:

Let’s Build an Incremental Source Generator With Roslyn, by Stefan Pölz
Vaporizing plastics recycles them into nothing but gas
OpenHCL: Evolving Azure’s virtualization model

Azure Boost is a revolutionary accelerator system designed by Microsoft that offloads server virtualization processes traditionally performed by the hypervisor and host OS onto purpose-built software and hardware. This offloading frees up CPU resources for virtual machines, resulting in improved performance and a secure foundation for your cloud workloads.
Going Super Sonic with Asio: Gotta go fast! Lessons learned for squeezing the most out of Asio for your application’s networking.
Grid-scale batteries: They’re not just lithium

When size and weight don’t matter, lots of other battery chemistries can work.

Monday 2024-09-23 Assorted Links

Mon, 23 Sep 2024 00:00:00 +0000

Today is C++ instrumentation day!

Assorted links for Monday, September 23:

Friday 2024-09-20 Assorted Links

Fri, 20 Sep 2024 00:00:00 +0000

Assorted links for Friday, September 20:

Thursday 2024-09-19 Assorted Links

Thu, 19 Sep 2024 00:00:00 +0000

Assorted links for Thursday, September 19:

Legacy Modernization meets GenAI

So far, most attention to Generative Artificial Intelligence (GenAI) in software development is on generating code. But we believe there is as much, if not more, value in understanding existing code - particularly long-lived, large, and complex legacy systems.
User-space interrupts

The term “interrupt” brings to mind a signal that originates in the hardware and which is handled in the kernel; even software interrupts are a kernel concept. But there is, it seems, a use case for enabling user-space processes to send interrupts directly to each other.

At its core, Mehta began, the user-space interrupts (or simply “user interrupts”) feature is a fast way to do event signaling. It delivers signals directly to user space, bypassing the kernel to achieve lower latency.
Dissecting the GZIP format
Storing and retrieving millions of ad impressions per second
Sacrificial architecture: Learning from abandoned systems

Wednesday 2024-09-18 Assorted Links

Wed, 18 Sep 2024 00:00:00 +0000

Assorted links for Wednesday, September 18:

As quantum computing threats loom, Microsoft updates its core crypto library

Microsoft has updated a key cryptographic library with two new encryption algorithms designed to withstand attacks from quantum computers.
OpenAI’s new “reasoning” AI models are here: o1-preview and o1-mini

OpenAI finally unveiled its rumored “Strawberry” AI language model on Thursday, claiming significant improvements in what it calls “reasoning” and problem-solving capabilities over previous large language models (LLMs). Formally named “OpenAI o1,” the model family will initially launch in two forms, o1-preview and o1-mini, available today for ChatGPT Plus and certain API users.
The Good and Bad of C++ As a Rust Dev
Secure by Design for AI: Building Resilient Systems from the Ground Up
Mice made transparent with a dye used in Doritos

But now, a team of Stanford University scientists has finally found an agent that can reversibly make skin transparent without damaging it. This agent was tartrazine, a popular yellow-orange food dye called FD&C Yellow 5 that is notably used for coloring Doritos.

Tuesday 2024-09-17 Assorted Links

Tue, 17 Sep 2024 00:00:00 +0000

Today is a .NET focused day.

Assorted links for Tuesday, September 17:

Performance Improvements in .NET 9
How Async/Await Really Works in C#
Dynamically Adapting To Application Sizes: A blog post about advanced garbage collection techniques.
Preventing breaking changes in public APIs with PublicApiGenerator. See also Preventing breaking changes in .NET class libraries.
A Complete .NET Developer’s Guide to Span with Stephen Toub

Monday 2024-09-16 Assorted Links

Mon, 16 Sep 2024 00:00:00 +0000

Assorted links for Monday, September 16:

Friday 2024-09-13 Assorted Links

Fri, 13 Sep 2024 00:00:00 +0000

Assorted links for Friday, September 13:

Thursday 2024-09-12 Assorted Links

Thu, 12 Sep 2024 00:00:00 +0000

Assorted links for Thursday, September 12:

noexcept affects libstdc++’s unordered_set: It may be worthwhile to investigate alternative data structure implementations like Boost.Unordered

GNU libstdc++’s hash-based associative containers change the struct layout of their nodes depending on the noexceptness of your hash function
RETINAS: Real-Time Infrastructure Accounting for Sustainability

We are introducing a new metric – real-time server fleet utilization effectiveness —- as part of the RETINAS initiative to help reduce emissions and achieve net zero emissions across our value chain in 2030. This new metric allows us to measure server resource usage (e.g., compute, storage) and efficiency in our large-scale data center server fleet in near real-time.
How Meta enforces purpose limitation via Privacy Aware Infrastructure at scale

Privacy Aware Infrastructure (PAI) offers efficient and reliable first-class privacy constructs embedded in Meta infrastructure to address complex privacy issues. For example, we built Policy Zones that apply across our infrastructure to address restrictions on data, such as using it only for allowed purposes, providing strong guarantees for limiting the purposes of its processing.
Landlock: unprivileged access control: Landlock is a userspace API provided by the Linux kernel.

The goal of Landlock is to enable to restrict ambient rights (e.g. global filesystem or network access) for a set of processes.
Intel Further Speeds Up strnlen() In The GNU C Library For Recent Intel/AMD CPUs: It’s incredible that we’re still finding performance improvements in venerable functions like strlen().

Wednesday 2024-09-11 Assorted Links

Wed, 11 Sep 2024 00:00:00 +0000

Assorted links for Wednesday, September 11:

Tuesday 2024-09-10 Assorted Links

Tue, 10 Sep 2024 00:00:00 +0000

Assorted links for Tuesday, September 10:

Visualizing boost::unordered_map in GDB, with pretty-printer customization points
Is Telegram really an encrypted messaging app?

From your perspective as a user, an “encrypted messenger” ensures that each time you start a conversation, your messages will only be readable by the folks you intend to speak with… Telegram clearly fails to meet this stronger definition for a simple reason: it does not end-to-end encrypt conversations by default. If you want to use end-to-end encryption in Telegram, you must manually activate an optional end-to-end encryption feature called “Secret Chats” for every single private conversation you want to have.
Threat Monitoring: Best Practices for Security
FUSE Adding IDMAPPED Mounts Support In Linux 6.12
Australia Threatens to Force Companies to Break Encryption

Monday 2024-09-09 Assorted Links

Mon, 09 Sep 2024 00:00:00 +0000

Assorted links for Monday, September 9:

A Guide to Managed Backup and Disaster Recovery
Continuous reinvention: A brief history of block storage at AWS

While the much celebrated ideal of a “full stack engineer” is valuable, in deep and complex systems it’s often even more valuable to create cohorts of experts who can collaborate and get really creative across the entire stack and all their individual areas of depth.
Untangling Lifetimes: The Arena Allocator
Governing data products using fitness functions

Fitness functions are a powerful automated governance technique we’ve applied to data products within the context of a Data Mesh. Since data products serve as the foundational building blocks (architectural quanta) of a data mesh, ensuring robust governance around them significantly increases the chances of a successful data mesh transformation.
Honey, I shrunk {fmt}: bringing binary size to 14k and ditching the C++ runtime

Friday 2024-08-30 Assorted Links

Fri, 30 Aug 2024 00:00:00 +0000

Assorted links for Friday, August 30:

Reimplementing A Linux Rust Scheduler In eBPF Shows Very Promising Results

In conclusion, prototyping new schedulers in user-space using Rust and then re-implementing them in BPF can be an effective workflow for designing new specialized schedulers.
Introducing Docker Build Checks: Optimize Dockerfiles with Best Practices
Docker Scout Health Scores: Security Grading for Container Images in Your Docker Hub Repo
50 Years of Queries

A discussion of the evolution of the database industry over the last half century, and why the relational database concepts introduced by E. F. Codd have proven so resilient over several decades.
Operational and Analytic Data Cycles

The relationship between operational and analytic data systems, and the challenges facing their implementation and management.

Thursday 2024-08-29 Assorted Links

Thu, 29 Aug 2024 00:00:00 +0000

Assorted links for Thursday, August 29:

Wednesday 2024-08-28 Assorted Links

Wed, 28 Aug 2024 00:00:00 +0000

Assorted links for Wednesday, August 28:

How Meta animates AI-generated images at scale
New AI algorithm flags deepfakes with 98% accuracy — better than any other tool out there right now
More than 100K sites impacted by Polyfill supply chain attack – An important way to protect your supply chain is to lock your dependency to a specific version + hash (e.g. SHA256). This way the dependency can’t be backdoored without you detecting it.
Providing Security Updates to Automobile Software
NIST Releases First Post-Quantum Encryption Algorithms

Tuesday 2024-08-27 Assorted Links

Tue, 27 Aug 2024 00:00:00 +0000

Assorted links for Tuesday, August 27:

Monday 2024-08-26 Assorted Links

Mon, 26 Aug 2024 00:00:00 +0000

Assorted links for Monday, August 26:

Friday 2024-08-16 Assorted Links

Fri, 16 Aug 2024 00:00:00 +0000

Assorted links for Friday, August 16:

Thursday 2024-08-15 Assorted Links

Thu, 15 Aug 2024 00:00:00 +0000

Assorted links for Thursday, August 15:

Wednesday 2024-08-14 Assorted Links

Wed, 14 Aug 2024 00:00:00 +0000

Assorted links for Wednesday, August 14:

Tuesday 2024-08-13 Assorted Links

Tue, 13 Aug 2024 00:00:00 +0000

Assorted links for Tuesday, August 13:

Monday 2024-08-12 Assorted Links

Mon, 12 Aug 2024 00:00:00 +0000

Assorted links for Monday, August 12:

Adding .NET Aspire to your existing .NET apps

But .NET Aspire is not just about cutting-edge technology and green-field apps; it’s also about making your current applications more straightforward. With .NET Aspire, you can streamline the startup process, improve monitoring, and increase the reliability of your applications.
Announcing Rust 1.80.1
.NET Cancellation by Stephen Cleary:
Networking improvements in Windows Server 2025
Machine Learning in Content Moderation at Etsy

Wednesday 2024-08-07 Assorted Links

Wed, 07 Aug 2024 00:00:00 +0000

Assorted links for Wednesday, August 7:

US solar production soars by 25 percent in just one year
Securely design your applications and protect your sensitive data with VBS enclaves

Virtualization Based Security (VBS) is the core feature of Windows used to the high value secrets stored within Windows (e.g., Credential Guard). VBS utilizes the Hyper-V hypervisor to create an environment that is higher privileged than the rest of the system kernel. Like VM isolation, the hypervisor sets memory protections in the second level address tables and IOMMU tables to isolate this environment from the rest of the system kernel.
Apple signs on to Biden’s responsible AI guidelines

Apple now joins 15 other major tech companies, including Amazon, Anthropic, Google, Inflection, Meta, Microsoft and OpenAI, in committing to responsible AI development and rollout.
N3199: Improved __attribute__((cleanup)) Through defer: A proposal for a defer keyword in C. See also defer reference implementation for C.
The C++ Object Lifecycle

Tuesday 2024-08-06 Assorted Links

Tue, 06 Aug 2024 00:00:00 +0000

Assorted links for Tuesday, August 6:

Thursday 2024-08-01 Assorted Links

Thu, 01 Aug 2024 00:00:00 +0000

Assorted links for Thursday, August 1:

Wednesday 2024-07-31 Assorted Links

Wed, 31 Jul 2024 00:00:00 +0000

Assorted links for Wednesday, July 31:

Tuesday 2024-07-30 Assorted Links

Tue, 30 Jul 2024 00:00:00 +0000

Assorted links for Tuesday, July 30:

Add AI to Your .NET Apps Easily with Prompty
Google halts its 4-plus-year plan to turn off tracking cookies by default in Chrome
AT&T failed to test disastrous update that kicked all devices off network
No More Blue Fridays: It used to be that reliability was achieved by moving code out of kernel mode and into user mode. eBPF suggests an alternative: allow code to run in kernel mode, but in a sandbox.

Monday 2024-07-29 Assorted Links

Mon, 29 Jul 2024 00:00:00 +0000

Assorted links for Monday, July 29:

rr

rr aspires to be your primary C/C++ debugging tool for Linux, replacing — well, enhancing — gdb. You record a failure once, then debug the recording, deterministically, as many times as you want. The same execution is replayed every time.
How we improved availability through iterative simplification
Introducing Bazel 7.2’s Output Service protocol

One of the exciting new features in Bazel 7.2 is support for the Bazel Output Service which allows Bazel to lazily materialize outputs when you access them with normal filesystem operations. This allows you to maintain visibility to the entire output tree while still saving network bandwidth.
How to review code effectively: A GitHub staff engineer’s philosophy
What’s new in .NET Aspire 8.1 for cloud native developers!

Friday 2024-07-26 Assorted Links

Fri, 26 Jul 2024 00:00:00 +0000

Assorted links for Friday, July 26:

Thursday 2024-07-25 Assorted Links

Thu, 25 Jul 2024 00:00:00 +0000

Assorted links for Thursday, July 25:

Profiling in production to detect server bottlenecks
OpenSLO

OpenSLO is a service level objective (SLO) language that declaratively defines reliability and performance targets using a simple YAML specification.
Understanding Java Garbage Collection Logging: What Are GC Logs and How to Analyze Them
Using Kubernetes to rethink your system architecture and ease technical debt
DevOps Practices for Continuous Deployment

Wednesday 2024-07-24 Assorted Links

Wed, 24 Jul 2024 00:00:00 +0000

Assorted links for Wednesday, July 24:

Tuesday 2024-07-23 Assorted Links

Tue, 23 Jul 2024 00:00:00 +0000

Assorted links for Tuesday, July 23:

Automerge is a library of data structures for building collaborative applications
Maestro: Netflix’s Workflow Orchestrator

Maestro is a general-purpose, horizontally scalable workflow orchestrator designed to manage large-scale workflows such as data pipelines and machine learning model training pipelines. It oversees the entire lifecycle of a workflow, from start to finish, including retries, queuing, task distribution to compute engines, etc.
Key JVM Metrics to Monitor for Peak Java Application Performance
“THANOS” — Monitoring with Prometheus and Grafana
Reducing data transfer costs with a Docker registry cache

Monday 2024-07-22 Assorted Links

Mon, 22 Jul 2024 00:00:00 +0000

Assorted links for Monday, July 22:

Friday 2024-07-19 Assorted Links

Fri, 19 Jul 2024 00:00:00 +0000

Today is (Un)Happy Crowdstrike Day! Assorted links for Friday, July 19:

My immediate thoughts are as follows:

Monocultures are inherently fragile and an high-value attack target. This applies whether you’re talking about Windows running 95%+ of desktops, CrowdStrike running 50%+ of Fortune 500 computers, or all Cavendish bananas being genetic clones of each other – an attack on one can easily become an attack on all. Sometimes it pays to not make the same choice as everyone else. Think security through diversity.
All automatic software deployment processes must always use progressive deployment with metric-based success gates and a straightforward, regularly-tested rollback process. If you can’t meet these requirements, you don’t deserve the ability to deploy your software automatically.

Wednesday 2024-07-17 Assorted Links

Wed, 17 Jul 2024 00:00:00 +0000

Assorted links for Wednesday, July 17:

Tuesday 2024-07-16 Assorted Links

Tue, 16 Jul 2024 00:00:00 +0000

Assorted links for Tuesday, July 16:

Monday 2024-07-15 Assorted Links

Mon, 15 Jul 2024 00:00:00 +0000

Assorted links for Monday, July 15:

“We’re Living in a Nightmare:” Inside the Health Crisis of a Texas Bitcoin Town
Ubuntu 24.10 To Enable Frame Pointers For More Packages: tl;dr: don’t use -fomit-frame-pointer anymore.
State of Text Rendering 2024
Scalpers Work With Hackers to Liberate Ticketmaster’s ‘Non-Transferable’ Tickets
Exploring the challenges in creating an accessible sortable list (drag-and-drop)

Friday 2024-07-12 Assorted Links

Fri, 12 Jul 2024 00:00:00 +0000

Assorted links for Friday, July 12:

Thursday 2024-07-11 Assorted Links

Thu, 11 Jul 2024 00:00:00 +0000

Assorted links for Thursday, July 11:

Wednesday 2024-07-10 Assorted Links

Wed, 10 Jul 2024 00:00:00 +0000

Assorted links for Wednesday, July 10:

Tuesday 2024-07-09 Assorted Links

Tue, 09 Jul 2024 00:00:00 +0000

Assorted links for Tuesday, July 9:

Monday 2024-07-08 Assorted Links

Mon, 08 Jul 2024 00:00:00 +0000

Assorted links for Monday, July 8:

Friday 2024-07-05 Assorted Links

Fri, 05 Jul 2024 00:00:00 +0000

Assorted links for Friday, July 5:

Thursday 2024-07-04 Assorted Links

Thu, 04 Jul 2024 00:00:00 +0000

Assorted links for Thursday, July 4:

Wednesday 2024-07-03 Assorted Links

Wed, 03 Jul 2024 00:00:00 +0000

Assorted links for Wednesday, July 3:

Error and Transaction Handling in SQL Server: Part One – Jumpstart Error Handling
Error and Transaction Handling in SQL Server: Part Two – Commands and Mechanisms
Error and Transaction Handling in SQL Server: Part Three – Implementation
SLICK: Adopting SLOs for improved reliability

SLICK can help us locate metric and performance data regarding the reliability of a specific service just by knowing its name. It does this by building an index of onboarded services that link to dashboards with standard visualizations to analyze and assess the service reliability. So, with a single click, it becomes possible to know whether a service currently meets or doesn’t meet user expectations. We can then start asking why.
Using Admission Controllers to Detect Container Drift at Runtime

Tuesday 2024-07-02 Assorted Links

Tue, 02 Jul 2024 00:00:00 +0000

Assorted links for Tuesday, July 2:

How to Measure DevSecOps Success: Key Metrics Explained
Key DevSecOps metrics:
1. Number of security vulnerabilities over time
2. Compliance with security policies
“Energy-smart” bricks need less power to make, are better insulation

According to the RMIT researchers, “Brick kilns worldwide consume 375 million tonnes (~340 million metric tons) of coal in combustion annually, which is equivalent to 675 million tonnes of CO₂ emission (~612 million metric tons).” This exceeds the combined annual carbon dioxide emissions of 130 million passenger vehicles in the US.
Researchers upend AI status quo by eliminating matrix multiplication in LLMs

In the new paper, titled “Scalable MatMul-free Language Modeling,” the researchers describe creating a custom 2.7 billion parameter model without using MatMul ([matrix multiplication]) that features similar performance to conventional large language models (LLMs). They also demonstrate running a 1.3 billion parameter model at 23.8 tokens per second on a GPU that was accelerated by a custom-programmed FPGA chip that uses about 13 watts of power (not counting the GPU’s power draw). The implication is that a more efficient FPGA “paves the way for the development of more efficient and hardware-friendly architectures,” they write.
Enhancing Netflix Reliability with Service-Level Prioritized Load Shedding

We implemented a concurrency limiter within PlayAPI that prioritizes user-initiated requests over prefetch requests without physically sharding the two request handlers. This mechanism uses the partitioning functionality of the open source Netflix/concurrency-limits Java library.
Explaining generative language models to (almost) anyone

Monday 2024-07-01 Assorted Links

Mon, 01 Jul 2024 00:00:00 +0000

Assorted links for Monday, July 1:

The Danger of Atomic Operations

Most engineers reach for atomic operations in an attempt to produce some lock-free mechanism. Furthermore, programmers enjoy the intellectual puzzle of using atomic operations. Both of these lead to clever implementations which are almost always ill-advised and often incorrect.
What an SBOM can do for you
sched_ext: a BPF-extensible scheduler class (Part 1)

sched_ext allows you to write and run your custom process scheduler optimized for your target workloads and hardware architectures using BPF programs.
sched_ext: scheduler architecture and interfaces (Part 2)
Leveraging AI for efficient incident response

We’ve streamlined our investigations through a combination of heuristic-based retrieval and large language model (LLM)-based ranking to provide AI-assisted root cause analysis. During backtesting, this system has achieved promising results: 42% accuracy in identifying root causes for investigations at their creation time related to our web monorepo.

Friday 2024-06-28 Assorted Links

Fri, 28 Jun 2024 00:00:00 +0000

Assorted links for Friday, June 28:

IncludeOS

IncludeOS is a minimal unikernel operating system for C++ services running in the cloud and on real hardware. Starting a program with #include <os> will include a tiny operating system into your service during link-time.
Designing Uber
Designing Tinder
Fixing Performance Regressions Before they Happen
How we used C++20 to eliminate an entire class of runtime bugs

Thursday 2024-06-27 Assorted Links

Thu, 27 Jun 2024 00:00:00 +0000

Assorted links for Thursday, June 27:

Wednesday 2024-06-26 Assorted Links

Wed, 26 Jun 2024 00:00:00 +0000

Assorted links for Wednesday, June 26:

Speed Up Your CI/CD Pipeline with Change-Based Testing in a Yarn-Based Monorepo: I note that only building and testing what changed is one of the core value propositions of Bazel, but adopting Bazel often requires large investment in engineering and training.
What makes a good REST API?
How to use DORA metrics to improve software delivery
Don’t Get Lost in the Metrics Maze: A Practical Guide to SLOs, SLIs, Error Budgets, and Toil
Static B-Trees
In this section, we generalize the techniques we developed for binary search to static B-trees and accelerate them further using SIMD instructions. In particular, we develop two new implicit data structures:
- The first is based on the memory layout of a B-tree, and, depending on the array size, it is up to 8x faster than std::lower_bound while using the same space as the array and only requiring a permutation of its elements.
- The second is based on the memory layout of a B+ tree, and it is up to 15x faster than std::lower_bound while using just 6-7% more memory — or 6-7% of the memory if we can keep the original sorted array.

Tuesday 2024-06-25 Assorted Links

Tue, 25 Jun 2024 00:00:00 +0000

Assorted links for Tuesday, June 25:

Radioactive drugs strike cancer with precision

Pluvicto and Lutathera are both built around small protein sequences, known as peptides. These peptides specifically bind to target receptors on cancer cells—PSMA in the case of prostate cancer and somatostatin receptors in the case of Lutathera—and deliver radiation through the decay of unstable lutetium.

Administered via infusion into the bloodstream, these drugs circulate throughout the body until they firmly attach to the surfaces of tumor cells they encounter. Anchored at these target sites, the lutetium isotope then releases two types of radiation that aid in cancer treatment. The primary emission consists of beta particles, high-energy electrons capable of penetrating tumors and surrounding cells, tearing into DNA and causing damage that ultimately triggers cell death.
Amazon Exploring MM-Local Memory Allocations To Help With Current/Future Speculation Attacks

Back in 2019 after various speculation-based CPU vulnerabilities began coming to light, Amazon engineers proposed process-local memory allocations for hiding KVM secrets. They were striving for an alternative mitigation for vulnerabilities like L1TF by essentially providing some memory regions for kernel allocations out of view/access from other kernel code. Amazon engineers this week laid out a new proposal after five years of ongoing Linux kernel improvements for MM-local memory allocations for dealing with current and future speculation-based cross-process attacks.
TypeSpec: An API design language that either competes with, or augments, OpenAPI.
Optimize Kubernetes Pods’ Startup Time Using VolumeSnapshots: If your K8S application uses enormous, static data sources, using VolumeSnapshots may speed up its launch time significantly.
Building a GitOps CI/CD Pipeline with GitHub Actions (SOC 2)

Monday 2024-06-24 Assorted Links

Mon, 24 Jun 2024 00:00:00 +0000

Assorted links for Monday, June 24:

The time smart quotes prevented the entire Office division from committing code
Video annotator: a framework for efficiently building video classifiers using vision-language models and active learning

We introduce a novel framework, Video Annotator (VA), which leverages active learning techniques and zero-shot capabilities of large vision-language models to guide users to focus their efforts on progressively harder examples, enhancing the model’s sample efficiency and keeping costs low.

VA seamlessly integrates model building into the data annotation process, facilitating user validation of the model before deployment, therefore helping with building trust and fostering a sense of ownership. VA also supports a continuous annotation process, allowing users to rapidly deploy models, monitor their quality in production, and swiftly fix any edge cases by annotating a few more examples and deploying a new model version.
PVF: A novel metric for understanding AI systems’ vulnerability against SDCs in model parameters

Parameter vulnerability factor (PVF) is a novel metric we’ve introduced with the aim to standardize the quantification of AI model vulnerability against parameter corruptions.
Keeping main green in a monorepo
Researchers describe how to tell if ChatGPT is confabulating

…[T]he researchers focus on what they call semantic entropy. This evaluates all the statistically likely answers evaluated by the LLM and determines how many of them are semantically equivalent. If a large number all have the same meaning, then the LLM is likely uncertain about phrasing but has the right answer. If not, then it is presumably in a situation where it would be prone to confabulation and should be prevented from doing so.

Friday 2024-06-21 Assorted Links

Fri, 21 Jun 2024 00:00:00 +0000

Assorted links for Friday, June 21:

MLow: Meta’s low bitrate audio codec

After nearly two years of active development and testing, we are proud to announce Meta Low Bitrate audio codec, aka MLow, which achieves two-times-better quality than Opus (POLQA MOS 1.89 vs 3.9 @ 6kbps WB). Even more importantly, we are able to achieve this great quality while keeping MLow’s computational complexity 10 percent lower than that of Opus.
Unlocking the power of unstructured data with RAG

To make the most of their unstructured data, development teams are turning to retrieval-augmented generation, or RAG, a method for customizing large language models (LLMs). They can use RAG to keep LLMs up to date with organizational knowledge and the latest information available on the web. They can also use RAG and LLMs to surface and extract insights from unstructured data.
LXC vs. Docker: Which One Should You Use?

LXC is not typically used for application development but for scenarios requiring full OS functionality or direct hardware integration. Its ability to provide isolated and secure environments with minimal overhead makes it suitable for infrastructure virtualization where traditional VMs might be too resource-intensive.

Docker’s utility in supporting rapid development cycles and complex architectures makes it a valuable tool for developers aiming to improve efficiency and operational consistency in their projects.
AES-GCM and breaking it on nonce reuse
Next-Level Boilerplate: An Inside Look Into Our .Net Clean Architecture Repo

Clean architecture is a widely adopted opinionated way to structure your code and to separate the concerns of the application into layers. The main idea is to separate the business logic from the infrastructure and presentation layers.

Thursday 2024-06-20 Assorted Links

Thu, 20 Jun 2024 00:00:00 +0000

Assorted links for Thursday, June 20:

How we improved push processing on GitHub

A push triggers a Kafka event, which is fanned out via independent consumers to many isolated jobs that can process the event without worrying about any other consumers.
Leveraging Rust in High-Performance Web Services

Rust’s ownership model is a fundamental feature that enhances both speed and safety. Every value in Rust has a unique owner, responsible for its cleanup when it’s no longer needed. This eliminates the need for a garbage collector and ensures efficient memory management. The ownership rules are enforced at compile time, which means there’s no runtime overhead.
systemd 256 Released With run0, systemd-vpick, importctl & Other New Features
Maintaining large-scale AI capacity at Meta

Outside of special cases, Meta maintains its fleet of clusters using a technique called maintenance trains. This is used for all capacity, including compute and storage capacity. A small number of servers are taken out of production and maintained with all applicable upgrades. Trains provide the guarantee that all capacity minus one maintenance domain is up and running 24/7, thus providing capacity predictability. This is mandatory for all capacity that is used for online and recurring training.
How Meta trains large language models at scale

Wednesday 2024-06-19 Assorted Links

Wed, 19 Jun 2024 00:00:00 +0000

Assorted links for Wednesday, June 19:

Arm64 on GitHub Actions: Powering faster, more efficient build systems

Developers can now take advantage of Arm-based hardware hosted by GitHub to build and deploy their release assets anywhere Arm architecture is used. Best of all, these runners are priced at 37% less than our x64 Linux and Windows runners.
Develop Kubernetes Operators in Java without Breaking a Sweat
The Energy Footprint of Humans and Large Language Models

Assuming an 8-hour workday and considering 260 workdays per year brings the annual energy cost of one person’s hour of daily work to around 6 kWh[a].

Now for the energy cost of running an LLM. We have set a target of 250 words in an hour. LLMs generate tokens, parts of words, so if we use the standard ratio (for English) of 0.75 words per token, our target for one hour of work is around 333 tokens. Measurements with Llama 65B reported around 4 Joules per output token [4]. This leads to 1,332 Joules for 333 tokens, about 0.00037 kWh.
Microsoft is reworking Recall after researchers point out its security problems

Microsoft’s upcoming Recall feature in Windows 11 has generated a wave of controversy this week following early testing that revealed huge security holes. The initial version of Recall saves screenshots and a large plaintext database tracking everything that users do on their PCs, and in the current version of the feature, it’s trivially easy to steal and view that database and all of those screenshots for any user on a given PC, even if you don’t have administrator access. Recall also does little to nothing to redact sensitive information from its screenshots or that database.

First and most significantly, the company says that Recall will be opt-in by default, so users will need to decide to turn it on. It may seem like a small change, but many users never touch the defaults on their PCs, and for Recall to be grabbing all of that data by default definitely puts more users at risk of having their data stolen unawares.

The company also says it’s adding additional protections to Recall to make the data harder to access. You’ll need to enable Windows Hello to use Recall, and you’ll need to authenticate via Windows Hello (whether it’s a face-scanning camera, fingerprint sensor, or PIN) each time you want to open the Recall app to view your data.
Building Generative AI apps with .NET 8

Tuesday 2024-06-18 Assorted Links

Tue, 18 Jun 2024 00:00:00 +0000

Assorted links for Tuesday, June 18:

Composable data management at Meta

By providing a reusable, state-of-the-art execution engine that is engine- and dialect-agnostic (i.e, it can be integrated with any data system and extended to follow any SQL-dialect semantic), Velox quickly received attention from the open-source community. Beyond our initial collaborators from IBM/Ahana, Intel, and Voltron Data, today more than 200 individual collaborators from more than 20 companies around the world participate in Velox’s continued development.
New warp drive concept does twist space, doesn’t move us very fast

A team of physicists has discovered that it’s possible to build a real, actual, physical warp drive and not break any known rules of physics. One caveat: the vessel doing the warping can’t exceed the speed of light, so you’re not going to get anywhere interesting any time soon. But this research still represents an important advance in our understanding of gravity.
Biggest Windows 11 update in 2 years nearly finalized, enters Release Preview

Windows 11 24H2 includes an updated compiler, kernel, and scheduler, all lower-level system changes made at least in part to better support Arm-based PCs. Existing Windows-on-Arm systems should also see a 10 or 20 percent performance boost when using x86 applications, thanks to improvements in the translation layer (which Microsoft is now calling Prism).

There are more user-visible changes, too. 24H2 includes Sudo for Windows, the ability to create TAR and 7-zip archives from the File Explorer, Wi-Fi 7 support, a new “energy saver” mode, and better support for Bluetooth Low Energy Audio. It also allows users to run the Copilot AI chatbot in a regular resizable window that can be pinned to the taskbar instead of always giving it a dedicated strip of screen space.
BitKeeper, Linux, and licensing disputes: How Linus wrote Git in 14 days
Another US state repeals law that protected ISPs from municipal competition

Minnesota this week eliminated two laws that made it harder for cities and towns to build their own broadband networks. The state-imposed restrictions were repealed in an omnibus commerce policy bill signed on Tuesday by Gov. Tim Walz, a Democrat.

Minnesota was previously one of about 20 states that imposed significant restrictions on municipal broadband. The number can differ depending on who’s counting because of disagreements over what counts as a significant restriction. But the list has gotten smaller in recent years because states including Arkansas, Colorado, and Washington repealed laws that hindered municipal broadband.

The Minnesota bill enacted this week struck down a requirement that municipal telecommunications networks be approved in an election with 65 percent of the vote. The law is over a century old, the Institute for Local Self-Reliance’s Community Broadband Network Initiative wrote yesterday.

Monday 2024-06-17 Assorted Links

Mon, 17 Jun 2024 00:00:00 +0000

Assorted links for Monday, June 17:

General Availability of .NET Aspire: Simplifying .NET Cloud-Native Development

.NET Aspire brings together tools, templates, and NuGet packages that help you build distributed applications in .NET more easily.
.NET Announcements and Updates from Microsoft Build 2024
Here’s a look at our updates & announcements:
- Artificial Intelligence: End-to-end scenarios for building AI-enabled applications, embracing the AI ecosystem, and deep integration with cloud services.
- .NET Aspire: for building cloud-native distributed applications, releasing today.
- C# 13: Improvements to much loved C# features to make them even better for you.
- Performance: Reducing memory and execution time with critical benchmarks.
- Enhancements to .NET libraries and frameworks including ASP.NET Core, Blazor, .NET MAUI, and more.
We get more useful energy out of renewables than fossil fuels

A new study by researchers at the UK’s University of Leeds, however, suggests that … renewables already produce more net energy than the fossil fuels they’re displacing. The key to understanding why is that it’s much easier to do useful things with electricity than it is with a hunk of coal or a glob of crude oil.
Docker Documentation Gets an AI-Powered Assistant

We recently launched a new tool to enhance Docker documentation: an AI-powered documentation assistant incorporating kapa.ai. Docker Docs AI is designed to get you the information you need by providing instant, accurate answers to your Docker-related questions directly within our documentation pages.
FUSE Adds VirtIO-FS Multi-Queue For ~5x Performance Win With Linux 6.10

With making use of multiple queues, the VirtIO-FS file-system code can be up to 5~5.5x faster for read and write performance.

Friday 2024-06-14 Assorted Links

Fri, 14 Jun 2024 00:00:00 +0000

Assorted links for Friday, June 14:

Updated Intel Meteor Lake Tuning For Linux Shows Huge Performance/Power Improvements. It is a minor tweak to the default Energy Performance Preference (EPP) value within the Intel P-State CPU frequency scaling driver.

It’s like magic with one line of code changed in the Linux kernel that Intel is reporting up to 19% performance improvement for Intel Core Ultra “Meteor Lake” and up to an 11% improvement in performance per Watt. Or in another EPP mode, the power consumption during video playback can be reduced by 52%!
These light paintings let us visualize invisible clouds of air pollution

Light painting is a technique used in both art and science that involves taking long-exposure photographs while moving some kind of light source—a small flashlight, perhaps, or candles or glowsticks—to essentially trace an image with light. A UK collaboration of scientists and artists has combined light painting with low-cost air pollution sensors to visualize concentrations of particulate matter (PM) in select locations in India, Ethiopia, and Wales. The objective is to creatively highlight the health risks posed by air pollution, according to a new paper published in the journal Nature Communications.
GPT-4 beats psychologists on a new test of social intelligence

There were significant differences in SI between psychologists and AI’s ChatGPT-4 and Bing. ChatGPT-4 exceeded 100% of all the psychologists, and Bing outperformed 50% of PhD holders and 90% of bachelor’s holders. The differences in SI between Google Bard and bachelor students were not significant, whereas the differences with PhDs were significant; Where 90% of PhD holders excel on Google Bird.
Wasm vs. Docker: Performant, Secure, and Versatile Containers
Battery Arbitrage

NYTimes: Since 2020, California has installed more giant batteries than anywhere in the world apart from China. They can soak up excess solar power during the day and store it for use when it gets dark.

Those batteries play a pivotal role in California’s electric grid, partially replacing fossil fuels in the evening. Between 7 p.m. and 10 p.m. on April 30, for example, batteries supplied more than one-fifth of California’s electricity and, for a few minutes, pumped out 7,046 megawatts of electricity, akin to the output from seven large nuclear reactors.

Thursday 2024-06-13 Assorted Links

Thu, 13 Jun 2024 00:00:00 +0000

Assorted links for Thursday, June 13:

Microsoft Rolling Out New Windows Subsystem For Linux “WSL” Features For 2024

Windows Subsystem for Linux is now automatically releasing stored memory in WSL back for use by Windows. This automatic memory reclaim support is a great addition and makes Windows behave better especially for systems with limited amounts of RAM. Without this support in memory hungry situations like with Docker it was possible for WSL2 to exhaust all of the system’s physical memory.

Windows Subsystem for Linux has also enabled DNS tunneling by default for improved network support.

Meanwhile in experimental form is support for automatic disk reclaim and a new mirrored networking mode that provides for features like IPv6 support.
New York Stock Exchange says bizarre glitch that showed Berkshire Hathaway down 99.97% has been resolved. Reversibility is an important feature of financial systems which naive blockchain systems deliberately forego.

For nearly two hours, Berkshire Hathaway’s Class A shares were listed as trading at just $185.10 — a price that would represent a loss of 99.97%. Berkshire closed at $627,400 on Friday.

NYSE announced it has decided to “bust,” or cancel, all “erroneous” trades for Berkshire between 9:50 am ET and 9:51 am ET at or below $603,718.30. The exchange said that ruling is not eligible for appeal and indicated it could cancel other trades.
Microsoft Releases Azure Linux 3.0 Preview

Azure Linux 3.0 shifts from the aging Linux 5.15 kernel to the newer Linux 6.6 LTS kernel as well as significant updates to OpenSSL, systemd, Runc, and other components. Azure Linux 3.0 is also now defaulting to SELinux’s enforcing mode by default.
Announcing the official OpenAI library for .NET
Today, the OpenAI team released their first beta, version 2.0.0-beta.1, of the official OpenAI library for .NET. Features include:
- Support for the entire OpenAI API, including Assistants v2 and Chat Completions
- Support for GPT-4o, OpenAI’s latest flagship model
- Extensibility to enable the community to build libraries on top
- Sync and async APIs for ease of use and efficiency
- Access to streaming completions via IAsyncEnumerable<T>
We’ve just had a year in which every month was a record-setter

Yesterday, the European Union’s Copernicus Earth-monitoring service announced that we’ve now gone a full year where every single month has been the warmest version of that month since we’ve had enough instruments in place to track global temperatures.

Wednesday 2024-06-12 Assorted Links

Wed, 12 Jun 2024 00:00:00 +0000

Assorted links for Wednesday, June 12:

Fracking wastewater has “shocking” amount of clean-energy mineral lithium

A study from researchers at the National Energy Technology Laboratory shows the wastewater produced by Pennsylvania’s unconventional wells could contain enough lithium to meet 38 to 40 percent of current domestic consumption.
Catch Up on Microsoft Build 2024: Essential Sessions for .NET Developers
Highlights from Microsoft Build: Docker’s Innovations with AI and Windows on Arm
Google Cloud explains how it accidentally deleted a customer account. The Google blog post is entitled Sharing details on a recent incident impacting one of our customers which is a ridiculous understatement.

During the initial deployment of a Google Cloud VMware Engine (GCVE) Private Cloud for the customer using an internal tool, there was an inadvertent misconfiguration of the GCVE service by Google operators due to leaving a parameter blank. This had the unintended and then unknown consequence of defaulting the customer’s GCVE Private Cloud to a fixed term, with automatic deletion at the end of that period. The incident trigger and the downstream system behavior have both been corrected to ensure that this cannot happen again.
AI passes the restaurant review Turing test. We are rapidly entering an era where we won’t be able to believe anything – articles, photos, videos, voice recordings – is genuine and original. Are you prepared?

In a series of experiments for a new study, Kovács found that a panel of human testers was unable to distinguish between reviews written by humans and those written by GPT-4, the LLM powering the latest iteration of ChatGPT. In fact, they were more confident about the authenticity of AI-written reviews than they were about human-written reviews.

Tuesday 2024-06-11 Assorted Links

Tue, 11 Jun 2024 00:00:00 +0000

Assorted links for Tuesday, June 11:

Experimental Windows Containers Support for BuildKit Released in v0.13.0

BuildKit is a toolkit for converting source code to build artifacts (like container images) in an efficient, expressive, and repeatable manner.
OpenAI training its next major AI model, forms new safety committee

On Monday, OpenAI announced the formation of a new “Safety and Security Committee” to oversee risk management for its projects and operations. The announcement comes as the company says it has “recently begun” training its next frontier model, which it expects to bring the company closer to its goal of achieving artificial general intelligence (AGI), though some critics say AGI is farther off than we might think. It also comes as a reaction to two weeks of public setbacks for the company.
Canonical Developing “Flamenco” For Enhancing .NET Developer Experience On Ubuntu

“Flamenco is a CLI tool that helps toolchain developers manage many different package versions and releases from a single debian folder source tree.”
GitHub and JFrog partner to unify code and binaries for DevSecOps

Together, we’ve built an integration that includes intuitive navigation and traceability between source code and binaries, CI/CD with GitHub Actions and JFrog Artifactory, and a unified view of security findings across the software supply chain. By providing full control and visibility across the entire software supply chain, we are accelerating our joint vision of making developers’ lives easier and happier.
Amazon Cloud Traffic Is Suffocating Fedora’s Mirrors

A massive uptick in traffic to Fedora’s package mirrors is causing problems for the Linux distribution. Some five million additional systems have started putting additional strain on Fedora’s mirror resources since March and appear to be coming from Amazon’s cloud.

Monday 2024-06-10 Assorted Links

Mon, 10 Jun 2024 00:00:00 +0000

Assorted links for Monday, June 10:

Federal agency warns critical Linux vulnerability being actively exploited

The vulnerability, tracked as CVE-2024-1086 and carrying a severity rating of 7.8 out of a possible 10, allows people who have already gained a foothold inside an affected system to escalate their system privileges. It’s the result of a use-after-free error, a class of vulnerability that occurs in software written in the C and C++ languages when a process continues to access a memory location after it has been freed or deallocated. Use-after-free vulnerabilities can result in remote code or privilege escalation.

The vulnerability, which affects Linux kernel versions 5.14 through 6.6, resides in the NF_tables, a kernel component enabling the Netfilter, which in turn facilitates a variety of network operations, including packet filtering, network address [and port] translation (NA[P]T), packet logging, userspace packet queueing, and other packet mangling. It was patched in January, but as the CISA advisory indicates, some production systems have yet to install it. At the time this Ars post went live, there were no known details about the active exploitation.
Google’s AI Overview is flawed by design, and a new company blog post hints at why

Here we see the fundamental flaw of the system: “AI Overviews are built to only show information that is backed up by top web results.” The design is based on the false assumption that Google’s page-ranking algorithm favors accurate results and not SEO-gamed garbage. Google Search has been broken for some time, and now the company is relying on those gamed and spam-filled results to feed its new AI model.
Online Privacy and Overfishing

Internet surveillance, and the resultant loss of privacy, is following the same trajectory. Just as certain fish populations in the world’s oceans have fallen 80 percent, from previously having fallen 80 percent, from previously having fallen 80 percent (ad infinitum), our expectations of privacy have similarly fallen precipitously. The pervasive nature of modern technology makes surveillance easier than ever before, while each successive generation of the public is accustomed to the privacy status quo of their youth. What seems normal to us in the security community is whatever was commonplace at the beginning of our careers.
The Danish Mortgage System Avoids Lock-In

Recall that in the Danish system each mortgage is backed by a matching bond. As a consequence, mortgage holders have two ways to pay a mortgage: 1) hold the mortgage and pay the monthly payments or 2) buy the matching bond and, in effect, extinguish the mortgage. The latter option is valuable because when interest rates rise, the price of mortgages fall.

…Danish sellers are able to earn a profit when they trade in their low mortgage rates for more-expensive ones, making it easier to move even when rates rise.
Vaccines don’t cause autism, but the lie won’t die. In fact, it’s getting worse.

In all, it’s a bleak finding that bodes poorly for the collective health of Americans, who are now seeing rises in cases of measles and other vaccine-preventable illnesses. Additional surveys by the APPC in 2021, 2022, and 2023 identified a slight increase in the number of survey takers who specifically believe, falsely, that the MMR (measles, mumps, and rubella) vaccine causes autism. In 2021, 9 percent of respondents falsely indicated that MMR vaccine causes autism, responding that the statement was “definitely true” (2 percent) or “probably true” (7 percent). In 2023, 12 percent of respondents fell into those categories, 2 percent for “definitely true” and 10 percent for “probably true.”

Since the start of 2024, the US has seen a steady march of measles infections nationwide. As of May 31, the CDC has recorded 146 cases across 21 states. Of those cases, 64 were part of a large outbreak in Chicago, which was declared over on May 30.

Wednesday 2024-05-22 Assorted Links

Wed, 22 May 2024 00:00:00 +0000

Assorted links for Wednesday, May 22:

Tuesday 2024-05-21 Assorted Links

Tue, 21 May 2024 00:00:00 +0000

Assorted links for Tuesday, May 21:

Monday 2024-05-20 Assorted Links

Mon, 20 May 2024 00:00:00 +0000

Assorted links for Monday, May 20:

Friday 2024-05-10 Assorted Links

Fri, 10 May 2024 00:00:00 +0000

Assorted links for Friday, May 10:

Thursday 2024-05-09 Assorted Links

Thu, 09 May 2024 00:00:00 +0000

Assorted links for Thursday, May 9:

Dependabot on GitHub Actions and self-hosted runners is now generally available
Speeding up C++ build times
Why OpenAI Replaced ChatGPT Plugins With GPTs
Making an Application More Robust With Data Mapping
The business of wallets

Regulators strongly prefer that deposits stay within the regulated banking sector. The single largest reason is that they’re worried that households’ immediately accessible stored funds stay safe and accessible. A major follow-up reason, less understood by non-specialists, is that regulated banks are bound to a long list of consumer protection items on the transaction level, not the institution level. A lot of the abuse in the economy happens in $50 and $5,000 increments, rather than multi-billion dollar increments. Regulators sleep happier knowing that this abuse happens at companies with teams of operators standing. Those operators will groan and chalk a disputed transaction, instance of fraud, or glitch in the matrix up to the operational losses budget rather than sticking a user with it.
How we built Text-to-SQL at Pinterest
Minimizing on-call burnout through alerts observability
What Is Amazon Resource Name (ARN)?
OpenTelemetry Best Practices #1: Naming
OpenTelemetry Best Practices #2 Agents, Sidecars, Collectors, Coded Instrumentation

Wednesday 2024-05-08 Assorted Links

Wed, 08 May 2024 00:00:00 +0000

Assorted links for Wednesday, May 8:

Development Containers Simplified
Highlights from Git 2.45
- If you are interested in how Git stores data behind the scenes, here is the reftable storage specification for the new reftable storage backend
systemd Rolling Out run0 As sudo Alternative: From what I can tell from the summary, run0 should be more secure than sudo
- Detailed commentary from Lennart Poterring is here
What we need to take away from the XZ Backdoor
Get started with the latest updates for Dockerfile syntax (v1.7.0)
MSVC Address Sanitizer adoption in .NET’s CoreCLR: I want to use significantly more static and dynamic analysis in my occasional C++ use
Visual Studio features for C++ Header Files and Modules
How to audit data modifications with surgical precision
Storing energy with compressed air is about to have its moment of truth
Introducing Artifact Attestations–now in public beta

Tuesday 2024-05-07 Assorted Links

Tue, 07 May 2024 00:00:00 +0000

Assorted links for Tuesday, May 7:

Streamline your container build and publish with .NET 8
Building a scalable gateway with .NET for Microsoft AI
Profiling Docker Compose services through Visual Studio
Refactor your C# code with primary constructors
What’s new for .NET in Ubuntu 24.04
Secure your container build and publish with .NET 8
Silent, brokered authentication on Windows with the Azure Identity libraries
Vendor-Locked DevOps Strategies: Personally I think vendor lock-in is mostly overblown.
4 ways GitHub engineers use GitHub Copilot
The Performance Impact of C++’s final Keyword: tl;dr, final isn’t always faster, and modern CPU performance defies intuition so always test & measure.

Monday 2024-05-06 Assorted Links

Mon, 06 May 2024 00:00:00 +0000

Assorted links for Monday, May 6:

Friday 2024-04-26 Assorted Links

Fri, 26 Apr 2024 00:00:00 +0000

Assorted links for Friday, April 26:

Thursday 2024-04-25 Assorted Links

Thu, 25 Apr 2024 00:00:00 +0000

Assorted links for Thursday, April 25:

Wednesday 2024-04-24 Assorted Links

Wed, 24 Apr 2024 00:00:00 +0000

Assorted links for Wednesday, April 24:

Tuesday 2024-04-23 Assorted Links

Tue, 23 Apr 2024 00:00:00 +0000

Assorted links for Tuesday, April 23:

Monday 2024-04-22 Assorted Links

Mon, 22 Apr 2024 00:00:00 +0000

Assorted links for Monday, April 22:

Friday 2024-04-19 Assorted Links

Fri, 19 Apr 2024 00:00:00 +0000

Assorted links for Friday, April 19:

Thursday 2024-04-18 Assorted Links

Thu, 18 Apr 2024 00:00:00 +0000

Assorted links for Thursday, April 18:

Wednesday 2024-04-17 Assorted Links

Wed, 17 Apr 2024 00:00:00 +0000

Assorted links for Wednesday, April 17:

Tuesday 2024-04-16 Assorted Links

Tue, 16 Apr 2024 00:00:00 +0000

Assorted links for Tuesday, April 16:

Monday 2024-04-15 Assorted Links

Mon, 15 Apr 2024 00:00:00 +0000

Assorted links for Monday, April 15:

Friday 2024-04-12 Assorted Links

Fri, 12 Apr 2024 00:00:00 +0000

Assorted links for Friday, April 12:

Thursday 2024-04-11 Assorted Links

Thu, 11 Apr 2024 00:00:00 +0000

Assorted links for Thursday, April 11:

GNU Stow 2.4.0 released – I use stow whenever I’m installing custom-compiled software.
Security Vulnerability of HTML Emails

The email your manager received and forwarded to you was something completely innocent, such as a potential customer asking a few questions. All that email was supposed to achieve was being forwarded to you. However, the moment the email appeared in your inbox, it changed. The innocent pretext disappeared and the real phishing email became visible. A phishing email you had to trust because you knew the sender and they even confirmed that they had forwarded it to you.
Improved Alerting with Atlas Streaming Eval
Why choose sum types over exceptions?
Refresh vs. Long-lived Access Tokens
Why you should use io_uring for network I/O
Clocks and Causality - Ordering Events in Distributed Systems
Twitter’s Recommendation Algorithm
Introducing self-service SBOMs
Practical tips for rightsizing your Kubernetes workloads

Wednesday 2024-04-10 Assorted Links

Wed, 10 Apr 2024 00:00:00 +0000

Assorted links for Wednesday, April 10:

Tuesday 2024-04-09 Assorted Links

Tue, 09 Apr 2024 00:00:00 +0000

Assorted links for Tuesday, April 9:

Monday 2024-04-08 Assorted Links

Mon, 08 Apr 2024 00:00:00 +0000

Assorted links for Monday, April 8:

GNU Coreutils 9.5 Can Yield 10~20% Throughput Boost For cp, mv & cat Commands: The speed increase is by increasing the default block size from 128KiB to 256KiB. Anyone still working in 4KiB or 64KiB blocks? Time to increase!
Biden orders every US agency to appoint a chief AI officer

As chief AI officers, appointees will serve as senior advisers on AI initiatives, monitoring and inventorying all agency uses of AI. They must conduct risk assessments to consider whether any AI uses are impacting “safety, security, civil rights, civil liberties, privacy, democratic values, human rights, equal opportunities, worker well-being, access to critical resources and services, agency trust and credibility, and market competition,” OMB said.
Oregon Just Passed a Monumental ‘Right to Repair’ Law

[W]hat sets this Oregon law apart from the other three states is that is bans “parts pairing,” a term that describes when companies prevent unauthorized parts from functioning in their devices.
Bing on .NET 8: The Impact of Dynamic PGO: .NET continues to deliver significant performance improvements release after release, with near-perfect backwards compatibility.
Google says running AI models on phones is a huge RAM hog: Is it possible that the largest barrier to Artificial General Intelligence (AGI) will be the amount of computing resources (RAM, GPU, electricity, etc.) necessary to run it?
Why the Department of Justice Cares About Green Bubbles on iPhones
Garbage Collection for Systems Programmers
Making AI powered .NET apps more consistent and intelligent with Redis
Improvements to static analysis in the GCC 14 compiler
Running local tools installed by Bazel

Friday 2024-04-05 Assorted Links

Fri, 05 Apr 2024 00:00:00 +0000

Assorted links for Friday, April 5:

Scaling up the Prime Video audio/video monitoring service and reducing costs by 90% – a case study of moving from distributed microservices back to a monolith application
Shaping live sports publishing traffic through a distributed scheduling system
Debugging a FUSE deadlock in the Linux kernel
Enum Class Bitmasks
Bazel in CI (Part 1): Commit Under Test
Bazel in CI (Part 2): Worker Setup
Bazel Caching Explained (pt. 3): Repository Cache
Fast builds, secure builds. Choose two. – How Stripe uses Bazel for build and test pipelines.
Bazel roadmap
Introducing rules_oci – A replacement for rules_docker which builds container images.

Thursday 2024-04-04 Assorted Links

Thu, 04 Apr 2024 00:00:00 +0000

Assorted links for Thursday, April 4:

Wednesday 2024-04-03 Assorted Links

Wed, 03 Apr 2024 00:00:00 +0000

Assorted links for Wednesday, April 3:

Tuesday 2024-04-02 Assorted Links

Tue, 02 Apr 2024 00:00:00 +0000

Assorted links for Tuesday, April 2:

Monday 2024-04-01 Assorted Links

Mon, 01 Apr 2024 00:00:00 +0000

Assorted links for Monday, April 1:

Thousands of servers hacked in ongoing attack targeting Ray AI framework: Researchers say it’s the first known in-the-wild attack targeting AI workloads.
An Alerting strategy for the cloud
Parsing Protobuf at 2+GB/s: How I Learned To Love Tail Calls in C
Advances in document understanding
From U2F to passkeys
Protecting Secrets with Docker
Improve Docker Compose Modularity with include
Container Security and Why It Matters
vcpkg integration with the GitHub dependency graph
Introducing Immortal Objects for Python

Friday 2024-03-29 Assorted Links

Fri, 29 Mar 2024 00:00:00 +0000

Assorted links for Friday, March 29:

Thursday 2024-03-28 Assorted Links

Thu, 28 Mar 2024 00:00:00 +0000

Assorted links for Thursday, March 28:

The RedMonk Programming Language Rankings: January 2024
How Figma’s databases team lived to tell the scale
Global products, global monitoring: Terraform Strategies for Seamless Grafana Dashboards Across Regions – this feels like using Terraform as the core of a GitOps strategy, which doesn’t sound like too bad of an idea to me
Licensing AI Engineers
TDD: You’re Probably Doing It Just Fine
Governments Setting Limits on AI: Many countries and regions are considering, or trying to implement, regulations on the training and use of artificial intelligence
Combining Machine Learning and Lifetime-Based Resource Management for Memory Allocation and Beyond
How to use GitHub Copilot in your IDE: Tips, tricks, and best practices
Unpatchable vulnerability in Apple chip leaks secret encryption keys
Use the ‘Hawthorne Effect’ to Increase Your Productivity

Wednesday 2024-03-27 Assorted Links

Wed, 27 Mar 2024 00:00:00 +0000

Assorted links for Wednesday, March 27:

SDL Storage API Merged For SDL 3.0
The Return of the Frame Pointers
Sequential A/B Testing Keeps the World Streaming Netflix Part 2: Counting Processes
Logarithm: A logging engine for AI training workflows and services
AI and the Evolution of Social Media
GoFetch: Breaking Constant-Time Cryptographic Implementations Using Data Memory-Dependent Prefetchers
Found means fixed: Introducing code scanning autofix, powered by GitHub Copilot and CodeQL
UN General Assembly adopts landmark resolution on artificial intelligence – sadly, this is a nonbinding agreement and thus effectively toothless
Linux Crisis Tools – install these Linux tools on your production servers before a crisis hits
The Danger of Unreliable Platforms (with Jade Rubick)

Tuesday 2024-03-26 Assorted Links

Tue, 26 Mar 2024 00:00:00 +0000

Assorted links for Tuesday, March 26:

Monday 2024-03-25 Assorted Links

Mon, 25 Mar 2024 00:00:00 +0000

Assorted links for Monday, March 25:

Friday 2024-03-22 Assorted Links

Fri, 22 Mar 2024 00:00:00 +0000

Assorted links for Friday, March 22:

Thursday 2024-03-21 Assorted Links

Thu, 21 Mar 2024 00:00:00 +0000

Assorted links for Thursday, March 21:

Wednesday 2024-03-20 Assorted Links

Wed, 20 Mar 2024 00:00:00 +0000

Assorted links for Wednesday, March 20:

Tuesday 2024-03-19 Assorted Links

Tue, 19 Mar 2024 00:00:00 +0000

Assorted links for Tuesday, March 19:

Platform Engineering: Orchestrating Applications, Platforms, and Infrastructure
Recent troubleshooting cases from our SREs, part 5. An unexpected crash due to unrelated software changes
Elevating Microservices Testing and Development using DynamicEnv
Code Review Security Checklist
OpenSSF Malicious Packages – A collection of reports of malicious packages identified in Open Source package repositories, consumable via the Open Source Vulnerability (OSV) format.
11 Small Search Platforms: Powerful Alternatives to Elasticsearch, OpenSearch, and Solr
Migrating Netflix to GraphQL Safely
Monitoring is a Pain
Classifying Event Driven Architecture
How to Manage Schemas and Handle Standardization

Monday 2024-03-18 Assorted Links

Mon, 18 Mar 2024 00:00:00 +0000

Assorted links for Monday, March 18:

This Clock Made Power Grids Possible
How we applied advanced fuzzing techniques to cURL
GitHub is enabling secret push protection for all users. This means that, by default, GitHub will detect and help prevent developers from accidentally committing secrets to repositories.
White House: Future Software Should be Memory Safe. The pressure on C/C++ continues to grow.
AI-generated articles prompt Wikipedia to downgrade CNET’s reliability rating. I did not realize that CNET was posting AI-written articles, but I am not shocked.
Solving Cold Start Timeouts with AWS Lambda and API Gateway
Transactions in a Microservice World
Service Delivery Index: A Driver for Reliability
From Blind Spots to Clear Insights: The Evolution of Observability Tools and Practices at Greenlight
Best practices for monitoring static web applications

Friday 2024-03-15 Assorted Links

Fri, 15 Mar 2024 00:00:00 +0000

Assorted links for Friday, March 15:

Thursday 2024-03-14 Assorted Links

Thu, 14 Mar 2024 00:00:00 +0000

Assorted links for Thursday, March 14:

Wednesday 2024-03-13 Assorted Links

Wed, 13 Mar 2024 00:00:00 +0000

Assorted links for Wednesday, March 13:

Tuesday 2024-03-12 Assorted Links

Tue, 12 Mar 2024 00:00:00 +0000

Assorted links for Tuesday, March 12:

Monday 2024-03-11 Assorted Links

Mon, 11 Mar 2024 00:00:00 +0000

Assorted links for Monday, March 11:

Friday 2024-03-08 Assorted Links

Fri, 08 Mar 2024 00:00:00 +0000

Assorted links for Friday, March 8:

Thursday 2024-03-07 Assorted Links

Thu, 07 Mar 2024 00:00:00 +0000

Assorted links for Thursday, March 7:

Wednesday 2024-03-06 Assorted Links

Wed, 06 Mar 2024 00:00:00 +0000

Assorted links for Wednesday, March 6:

Tuesday 2024-03-05 Assorted Links

Tue, 05 Mar 2024 00:00:00 +0000

Assorted links for Tuesday, March 5:

Monday 2024-03-04 Assorted Links

Mon, 04 Mar 2024 00:00:00 +0000

Assorted links for Monday, March 4:

Friday 2024-03-01 Assorted Links

Fri, 01 Mar 2024 00:00:00 +0000

Assorted links for Friday, March 1:

Thursday 2024-02-29 Assorted Links

Thu, 29 Feb 2024 00:00:00 +0000

Assorted links for Thursday, February 29:

Wednesday 2024-02-28 Assorted Links

Wed, 28 Feb 2024 00:00:00 +0000

Assorted links for Wednesday, February 28:

Tuesday 2024-02-27 Assorted Links

Tue, 27 Feb 2024 00:00:00 +0000

Assorted links for Tuesday, February 27:

Monday 2024-02-26 Assorted Links

Mon, 26 Feb 2024 00:00:00 +0000

Assorted links for Monday, February 26:

Practical Bazel: Downloading Private Release Assets from GitHub

Tue, 23 May 2023 00:00:00 +0000

Downloading a private release asset from GitHub given only its name and tag requires a complicated series of interactions with the GitHub API. This blog post explains how to write two repository rules which make dealing with private release assets in Bazel easy.

Practical Bazel: Use multirun for parallel execution

Thu, 09 Mar 2023 00:00:00 +0000

A quick tip on how to execute multiple run commands in parallel.

Practical Bazel: Using Clang 12 C++ Toolchain on Ubuntu 20.04

Wed, 08 Mar 2023 00:00:00 +0000

This blog post explains how to use a custom C++ toolchain based on clang 12 with libc++-12 in Bazel on Ubuntu 20.04.

Practical Bazel: Testing Rulesets with Docker Compose-Based Buildfarm

Tue, 07 Mar 2023 00:00:00 +0000

Bazel supports scaling out builds with a remote execution system. Unfortunately, it is very easy for ruleset authors to release rules that work when executed locally but do not work when executed remotely. This blog post explains ruleset authors can set up a simple remote execution system to verify that their rulesets work correctly.

Practical Bazel: Supporting Trinary Stamp Attributes

Mon, 06 Mar 2023 00:00:00 +0000

In Bazel, stamping is the process of embedding additional information into built binaries, such as the source control revision or other workspace-related information. Rules that support stamping typically include an integer stamp attribute, where 1 means “always stamp”, 0 means “never stamp”, and -1 means “use the Bazel build --stamp flag. This blog post explains how to write a rule that supports these values.

Practical Bazel: Custom Bazel Make Variables

Fri, 03 Mar 2023 00:00:00 +0000

Many Bazel attributes support the use of predefined variables and functions such as @D for output directory or $(location //foo:bar) to get the path to a label. But what if you want to apply some sort of tranformation to these variables, or define your own custom make variables? This blog post explains how.

Practical Bazel: Retrieving Secrets from Netrc in Custom Rules

Thu, 02 Mar 2023 00:00:00 +0000

Bazel developers are currently working on adding the ability to retrieve secrets using a credential-helper executable, similar to how other tools like Docker and Git handle managing secrets. Until then, the recommended approach is to store secrets in ~/.netrc. This blog post explains how to write a custom Bazel rule which reads secrets from ~/.netrc.

Practical Bazel: local_archive() Workspace Rule

Wed, 01 Mar 2023 00:00:00 +0000

In general, one should never check in binary artifacts into Git; it is better to retrieve them from an artifact repository or a website using http_archive(). However, sometimes convenience is more important than ideological purity. To handle these cases, I wrote a simple workspace rule named local_archive().

Practical Bazel: Comparing Collections for Equivalence in sh_test()

Tue, 28 Feb 2023 00:00:00 +0000

When writing Bazel tests using sh_test(), I often find myself needing to compare two collections for equivalence. For example, I might compare a directory listing against a set of expected files or directories, or the list of files and directories in a .tar file against a set of expected items. This blog post provides some tips and tricks as to how to do so.

Practical Bazel: Using Bazel's Bash Runfiles Library

Mon, 27 Feb 2023 00:00:00 +0000

Dealing with Bazel runfiles is one of the most annoying things about using Bazel. Fortunately, Bazel provides a library to make resolving runfiles from Bash scripts easy.

Practical Bazel: Using Bazel's Bash Unittest Framework

Sun, 26 Feb 2023 00:00:00 +0000

sh_test is my most commonly used test rule by far. It is the easiest way to write quick-and-dirty tests and works nearly everywhere. For anything beyond the most trivial tests, I use Bazel’s Bash unit test framework. This explains what the framework is and how to use it.

Practical Bazel: Start with Official Rules Template

Fri, 18 Nov 2022 00:00:00 +0000

A quick tip for today:

When creating a new ruleset, particularly on GitHub, start with the official Bazel rules template. It includes a number of features out of the box that are rather tiresome to implement yourself.

Practical Bazel: Using Rulesets from Non-Public GitHub Repositories

Wed, 09 Nov 2022 00:00:00 +0000

At work, we have a number of custom-written Bazel rulesets stored in organization repositories on GitHub¹. This post explains how we use these non-public rulesets in our Bazel projects.

Practical Bazel: Building Multiple Workspaces in CI

Tue, 18 Oct 2022 00:00:00 +0000

For most Bazel projects, I strongly recommend using a single Bazel workspace per source code repository. However, it can be occasionally useful to nest multiple workspaces within a single repository. For example, when I’m writing Bazel rulesets, I will often create test cases that contain own workspace with a slightly different configuration in order to test various workspace-level configuration settings for the ruleset, while maintaining a root workspace which is the primary workspace for the ruleset.

Practical Bazel: Use a Single .bazelrc With Multiple Configurations

Mon, 17 Oct 2022 00:00:00 +0000

Bazel users commonly need to manage multiple different configurations when building and testing software, such as:

A configuration that is used by developers on their own PCs, which often has debug mode turned on
A configuration that is used by the continuous integration (CI) default build and test pipeline, which often builds in release mode
A configuration for generating a tagged, version release, which often has stamping enabled
(Perhaps many more)

Practical Bazel: Implementing compilation_mode in Rules

Thu, 13 Oct 2022 00:00:00 +0000

As developers know, virtually all compilers support the notion of compilation mode, e.g.. whether to compile the code in debug or release mode. In Bazel, this concept is built natively into the build system itself. When compiling software using bazel build, one can select a compilation mode using the --compilation_mode (often shorted to -c) flag.

Practical Bazel: Isolating Ruleset's Public Interface

Wed, 12 Oct 2022 00:00:00 +0000

When writing a custom Bazel ruleset, it is important to carefully separate its public interface from its private implementation and be deliberate and careful about changes to the public interface. Here’s the pattern I use when I’m writing rulesets to handle this.

Practical Bazel: Depending on a System-Provided C/C++ Library

Wed, 22 Sep 2021 00:00:00 +0000

Let’s say you are using Bazel to build a C program which links against a system-provided version of libcurl, the multiprotocol file transfer library. What is the best way to link your program against this library within Bazel? This blog post provides an answer to that question.

Practical Bazel: A Starting CI Pipeline

Tue, 14 Sep 2021 00:00:00 +0000

This post describes a pattern for implementing a continuous integration (CI) pipeline using Bazel. This pattern is my starting point whenever I set up a new Bazel-based project in CI, after which I add any project-specific pipeline customizations.

This pipeline is purely about the CI (build to release) stages of a pipeline. A full continuous delivery (CD) pipeline, which includes deployment, will be discussed in a later post.

Practical Bazel: A Successful Build is a Quiet Build

Sat, 04 Sep 2021 00:00:00 +0000

In Bazel, a successful build should be a quiet build. While build failures should, of course, print ample information to stderr to aide in troubleshooting, any custom Bazel code you write should not output progress information to stdout or stderr. Let Bazel be responsible for overall build progress reporting.

Practical Bazel: Use Awesome Bazel to find rules

Tue, 01 Dec 2020 00:00:00 +0000

A quick tip for today:

Use Awesome Bazel to find new Bazel rulesets, tooling, and resources. Don’t be afraid to submit PRs to its GitHub repo.

Practical Bazel: A Simpler Way to Wrap Run Targets

Fri, 20 Nov 2020 00:00:00 +0000

Yesterday, I explained how you can wrap a bazel run target with a sh_binary() to execute arbitrary code both before and after the run target, which is particularly useful for retrieving secrets from a secret management system and passing them to the run target.

If you are passing secrets via environment variables that are retrieved by command-line programs, there’s an even easier way to do it – use the command rule from Atlassian’s bazel-tools repo and its raw_environment attribute.

Practical Bazel: Wrapping Run Targets to Provide Additional Context

Thu, 19 Nov 2020 00:00:00 +0000

An executable rule which can be executed via bazel run is the natural way to model interactions with external systems in Bazel such as uploading build artifacts to a remote artifact repository. For example, imagine a rules_artifactory ruleset which includes a rule artifactory_push() executable rule which uploads a compiled .dpkg to an Artifactory apt repository, or a rules_docker ruleset which has a rule docker_push() which pushes a Docker image to a remote image repository.

Practical Bazel: Writing a Rule That Supports Both Build and Run

Fri, 13 Nov 2020 00:00:00 +0000

When I first started writing custom Bazel rules, I often created separate rules for the build and run commands in Bazel. This was a mistake.

Practical Bazel: Use ctx.actions.args() to Form Arguments Lists

Thu, 12 Nov 2020 00:00:00 +0000

When writing custom rules, you often need to invoke executables with argument lists. For example, let’s say you are writing a custom rule that executes gcc to compile a set of input source files. You could write:

Practical Bazel: Changing Behavior on Windows

Thu, 05 Nov 2020 00:00:00 +0000

Bazel started on Linux and Mac OS, and most people use Bazel on these platforms exclusively, but Bazel can execute on Windows as well. However, Windows has enough idiosynchatic differences that writing a single, operating-system agnostic rule that executes on both Windows and Linux/Mac is quite hard. Often it is easiest to have the rule detect whether it is running on Windows and execute different behavior.

Practical Bazel: Prefix All Generated Files

Wed, 04 Nov 2020 00:00:00 +0000

When writing a custom rule that generates files, be sure to add prefixes to all filenames so that multiple instances of your rule can be instantiated within the same Bazel package.

Practical Bazel: Enable Bash Strict Mode

Fri, 30 Oct 2020 00:00:00 +0000

One tends to write a lot of Bash scripts when using Bazel. In order to make these scripts more robust, enable Bash’s unofficial strict mode by starting all scripts like this:

Practical Bazel: Force Line Endings to LF

Thu, 29 Oct 2020 00:00:00 +0000

Bazel requires all files to end with LF (not CR-LF) in order to work correctly.

If you are performing cross-platform Bazel development with Windows users, you can force this setting for all text files in your repo by creating a .gitattributes file with the following content:

Practical Bazel: Excluding Part of Tree

Wed, 28 Oct 2020 00:00:00 +0000

Quick Bazel tip for today: If you want to build everything except for a specific subtree, you can prefix the subtree you want to exclude with a -.

For example, to build everything except for //client_access_library/...:

bazel build -- //... -//client_access_library/...

Practical Bazel: Use a Specific Version of Bazel

Tue, 27 Oct 2020 00:00:00 +0000

Bazel’s philosophy strongly encourages binding to exact, specific versions of all third-party dependencies to help ensure reproducible builds. As Bazel users, we must remember to extend this philosophy to Bazel itself.

When setting up a Bazel-based build system, you should choose a specific version of Bazel and require all developers and the build system to use it. This can be done in a few ways:

Use Bazelisk and a .bazelversion file (recommended)
Installing a specific version of Bazel as part of your build and developer VM images / docker containers

With the first approach, upgrades to Bazel can then be treated like any other upgrade: submit a pull request to the repo which updates the value in the .bazelversion file and runs the full continuous integration pipeline.

Practical Bazel: path or short_path?

Thu, 22 Oct 2020 00:00:00 +0000

When writing custom Bazel rules, you spend a lot of time either reading or writing Bazel File objects.

File objects have two properties for accessing the underlying file path: File.path and File.short_path. When writing custom rules, I often chose one of the two properties at random, and switched to the other if it didn’t work right.

I wrote some simple custom rules to test the various combination of rule types and file types to determine when I should use path or short_path. The custom rules used bash script templates that were populated with file paths, such as:

Practical Bazel: Start with Genrules

Wed, 21 Oct 2020 00:00:00 +0000

Bazel is a powerful yet complicated system, and it can be intimidating to newcomers.

While the Bazel user guide and user manual preach the benefits of giving Bazel full control over your build process by rewriting all build processes using Bazel-native rulesets (as Google reportedly does internally), this is an immense amount of work. Specifically, if you are integrating third-party software into your Bazel-based build process, reverse engineering and rewriting the third-party project’s build system into Bazel can easily take days – and then you need to maintain it.

Practical Bazel: Introduction

Tue, 20 Oct 2020 00:00:00 +0000

In 2020, I led the redesign and re-implementation of the object storage system behind RelativityOne.

As part of this project we reengineered the continuous delivery pipeline of the service to embrace the philosophy of a service-wide monorepo with a Bazel-based build system. We chose Bazel because we wanted a build system that could support many different languages (the service has code written in C, C#, Python, Go, Terraform, Packer, and other languages…) while remaining fast and correct.

COVID-19's Effect on Alcohol Sales

Mon, 19 Oct 2020 00:00:00 +0000

Alcohol sales in the United States showed a clear and predictable pattern year-over-year … until 2020.

I’ve certainly been doing my part to bend the curve.

Source: FRED’s series on Retail Sales: Beer, Wine, and Liquor Stores

Dialect Quiz Results

Fri, 16 Oct 2020 00:00:00 +0000

Here are the results I got when I took New York Times dialect quiz test:

Eerily accurate.

Implication of Copyrightable APIs

Thu, 15 Oct 2020 00:00:00 +0000

The Supreme Court’s eight justices on Wednesday seemed skeptical of Google' argument that application programming interfaces (APIs) are not protected by copyright law. The high court was hearing oral arguments in Google’s decade-long legal battle with Oracle. Oracle argues that Google infringed its copyright in the Java programming language when it re-implemented Java APIs for use by Android app developers.

The stakes in the case are high for Google, which could owe Oracle billions of dollars in damages. More importantly, an Oracle win could reshape how copyright law treats APIs, giving incumbents the power to lock out competitors who want to build compatible software.

Streaming Percentiles 3.1.0 Released

Fri, 29 Mar 2019 00:00:00 +0000

Version 3.1.0 of the streaming percentiles library has been released.

The streaming percentiles library is a cross-platform, multi-language (C++ and JavaScript) implementation of a number of online (single-pass) percentile algorithms. This version of the streaming percentiles library adds support for copy construction, assignment, move construction, and move assignment on all analytics classes.

This change allows you to put streaming analytics classes into STL containers, such as:

#include <vector>
#include <stmpct/gk.hpp>

std::vector<stmpct::gk<double>> algs;
algs.emplace_back(0.01);
algs.emplace_back(0.05);
for (auto& g : algs) {
    for (int i = 0; i < 1000; ++i) {
        g.insert(rand());
    }

    double p95 = g.quantile(0.95); // Approx. 95th percentile for current epsilon
}

It also allows you to efficiently move the state of an algorithm from one object to another without copying, as in:

Calculating Percentiles on Streaming Data Part 8: Parameterizing Algorithms on Measurement Type

Fri, 21 Dec 2018 00:00:00 +0000

This is part 8/8 of my Calculating Percentiles on Streaming Data series.

As mentioned in part 6 of this series, I have published a C++ and JavaScript library which implements a number of streaming percentile algorithms on GitHub at https://github.com/sengelha/streaming-percentiles.

Versions 1.x and 2.x of the C++ library required all measurements to use the type double, and usage of the algorithms looked something like this:

#include <stmpct/gk.hpp>

double epsilon = 0.1;
stmpct::gk g(epsilon);
for (int i = 0; i < 1000; ++i)
    g.insert(rand());
double p50 = g.quantile(0.5); // Approx. median
double p95 = g.quantile(0.95); // Approx. 95th percentile

For version 3.x of the library, I have modified the algorithms to be parameterized by type so that they are not limited to double. The algorithms now may be used by type which implements a few simple requirements: it must be default constructible, copy constructible, assignable, and implement operator<. This means that they can now be used with float, int, long double, a custom number type, or even std::string.

Streaming Percentiles Library

Sun, 21 Oct 2018 00:00:00 +0000

At various points in my career, I have created or managed systems which need to perform calculations on enormous amounts of numeric data in real-time. Whether I was running a Varnish cluster that served 30,000 requests per second or writing financial analytics which needed to calculate aggregates on terabytes of price and return data in a few seconds, a common need emerged: how can I calcalate aggregates on large amounts of numeric data in a single pass?

D3 Latency Heatmap

Fri, 19 Oct 2018 00:00:00 +0000

In the fall of 2017, I was researching a performance problem with a job queueing system. I was trying to figure out how I could better visualize the response time of the system in order to understand what was going on and I ran across the excellent article Brandon Gregg’s Latency Heat Maps. I could not find an easy-to-use, reusable latency heat map chart component, so I decided to make my own in d3.js.

Moved Link Aggregation to Bitly

Sun, 22 Apr 2018 00:00:00 +0000

I have decided to move my link aggregation efforts, such as they are, to Bitly, and to leave this blog for original content. Follow me on Twitter if you’re interested in tracking these links in the future.

Unit Testing Emscripten Library in Browser Using CMake and Nightwatch.JS

Tue, 17 Apr 2018 00:00:00 +0000

In a previous blog post, I described how I took Emscripten-created JS and turned it into an UMD module. One of the reasons I did this was because I wanted more control over the generated JavaScript and for it to be usable in more contexts, such as with the RequireJS module loader.

As I am a responsible developer, I desired to create a number of automated unit tests to ensure that the client-visible API for my Emscripten module works as I intended. I began by searching for a browser automated test framework and settled upon Nightwatch.js. Now I just had to figure out how to get Nightwatch.js tests running in my existing, CMake-based build system. Here’s how I did it.

Creating UMD Module from Emscripten using CMake

Sat, 14 Apr 2018 00:00:00 +0000

By default, Emscripten creates a module which can be used from both Node.JS and the browser, but it has the following issues:

The module pollutes the global namespace
The module is created with the name Module (in my case, I require streamingPercentiles)
The module cannot be loaded by some module loaders such as require.js

While the above issues can (mostly) be corrected by using –s MODULARIZE=1, it changes the semantics of the resulting JavaScript file, as the module now returns a function rather than an object. For example, code which previously read var x = new Module.Klass() would become var x = new Module().Klass(). I found this semantic change unacceptable, so I decided to abandon Emscripten’s -s MODULARIZE=1 option in favor of hand-crafting a UMD module.

Board Games for Family Night

Thu, 05 Apr 2018 00:00:00 +0000

Playing board games is a major hobby of mine. Every year I look forward to attending Gen Con, where I aggressively shop the dings and dents section of CoolStuffInc and play as many new games as I can.

We enjoy playing board games as a family. I’ve been playing board games with my daughter since she was four; she’s eight now, and quite the gamer. One of the things we’ve learned is that the recommended age range on a board game is merely a recommendation: the only real challenge for my daughter in playing advanced games was how well she was able to read.

Calculating Percentiles on Streaming Data Part 7: Cormode-Korn-Muthukrishnan-Srivastava

Thu, 29 Mar 2018 00:00:00 +0000

This is part 7/8 of my Calculating Percentiles on Streaming Data series.

In 2005, Graham Cormode, Flip Korn, S. Muthukrishnan, and Divesh Srivastava published a paper called Effective Computation of Biased Quantiles over Data Streams [CKMS05]. This paper took the Greenwald-Khanna algorithm [GK01] and made the following notable changes:

Generalized the algorithm to work with arbitrary targeted quantiles, where a targeted quantile is a combination of a quantile $\phi$ and a maximum allowable error $\epsilon$. The algorithm will maintain the summary data structure $\mathsf{S}$ such that it will maintain each targeted quantile with the appropriate error rate.
Simplified the Compress() operation by removing the concept of banding.

The paper also demonstrated that the new, more generalized algorithm could handle the following cases with ease:

Calculating Percentiles on Streaming Data Part 6: Building a C++ and JavaScript Library from a Single Codebase

Mon, 19 Mar 2018 00:00:00 +0000

This is part 6/8 of my Calculating Percentiles on Streaming Data series.

For the past 10 days or so, I’ve been working on the build process of my C++ and JavaScript streaming analytics libraries. Using the magic of Emscripten, I have been able to combine both libraries into a single, C++ codebase, from which I can compile both the C++ and JavaScript versions of the library. Furthermore, I was able to do this without breaking backwards compatibility of the JavaScript library. I also made a number of other improvements to the compilation process, such as providing precompiled binaries and supporting both shared and static libraries.

Calculating Percentiles on Streaming Data Part 5: C++ Library

Fri, 09 Mar 2018 00:00:00 +0000

This is part 5/8 of my Calculating Percentiles on Streaming Data series.

I have created a reusable C++ library which contains my implementation of the streaming percentile algorithms found within this blog post and published it to GitHub. Here’s what using it looks like:

#include <stmpct/gk.hpp>

using namespace stmpct;

double epsilon = 0.1;
gk g(epsilon);
for (int i = 0; i < 1000; ++i)
    g.insert(rand());
double p50 = g.quantile(0.5); // Approx. median
double p95 = g.quantile(0.95); // Approx. 95th percentile

You can find it here:

Source code on GitHub

Calculating Percentiles on Streaming Data Part 4: JavaScript Library

Thu, 08 Mar 2018 00:00:00 +0000

This is part 4/8 of my Calculating Percentiles on Streaming Data series.

I have created a reusable JavaScript library which contains my implementation of the streaming percentile algorithms found within this blog post and published it to GitHub and NPM. Here’s what using it looks like:

var sp = require("streaming-percentiles');

// epsilon is allowable error. As epsilon becomes smaller, the
// accuracy of the approximations improves, but the class consumes
// more memory.
var epsilon = 0.1;
var gk = new sp.GK(epsilon);
for (var i = 0; i < 1000; ++i)
    gk.insert(Math.random());
var p50 = gk.quantile(0.5); // Approx. median
var p95 = gk.quantile(0.95); // Approx. 95th percentile

You can find it here:

Calculating Percentiles on Streaming Data Part 3: Visualizing Greenwald-Khanna

Wed, 07 Mar 2018 23:37:38 +0000

This is part 3/8 of my Calculating Percentiles on Streaming Data series.

In an effort to better understand the Greenwald-Khanna [GK01] algorithm, I created a series of visualizations of the cumulative distribution functions of a randomly-generated, normally-distributed data set with $\mu$ = 0 and $\sigma$ = 1, as the number of random numbers $n$ increases from 1 to 1,000.

The way to read these visualizations is to find the percentile you are looking for on the y-axis, then trace horizontally to find the vertical line on the chart which intersects this location, then read the value from the x-axis.

Calculating Percentiles on Streaming Data Part 2: Notes on Implementing Greenwald-Khanna

Wed, 07 Mar 2018 16:19:29 +0000

This is part 2/8 of my Calculating Percentiles on Streaming Data series.

The most famous algorithm for calculating percentiles on streaming data appears to be Greenwald-Khanna [GK01]. I spent a few days implementing the Greenwald-Khanna algorithm from the paper and I discovered a few things I wanted to share.

Insert Operation

The insert operation is defined in [GK01] as follows:

INSERT($v$): Find the smallest $i$, such that $v_{i-1} \leq v < v_i$, and insert the tuple $ t_x = (v_x, g_x, \Delta_x) = (v, 1, \lfloor 2 \epsilon n \rfloor) $, between $t_{i-1}$ and $t_i$. Increment $s$. As a special case, if $v$ is the new minimum or the maximum observation seen, then insert $(v, 1, 0)$.

However, when I implemented this operation, I noticed via testing that there were certain queries I could not fulfill. For example, consider applying Greenwald-Khanna with $\epsilon = 0.1$ to the sequence of values ${11, 20, 18, 5, 12, 6, 3, 2}$, and then apply QUANTILE($\phi = 0.5$). This means that $r = \lceil \phi n \rceil = \lceil 0.5 \times 8 \rceil = 4$ and $\epsilon n = 0.1 \times 8 = 0.8$. There is no $i$ that satisfies both $r – r_{min}(v_i) \leq \epsilon n$ and $r_{max}(v_i) – r \leq \epsilon n$, as you can see below:

Calculating Percentiles on Streaming Data Part 1: Introduction

Tue, 06 Mar 2018 00:00:00 +0000

This is part 1/8 of my Calculating Percentiles on Streaming Data series.

Suppose that you are dealing with a system which processes one million requests per second, and you’d like to calculate the median percentile response time over the last 24 hours.

The naive approach would be to store every response time, sort them all, and then return the value in the middle. Unfortunately, this approach would require manipulating 1,000,000 * 60 * 60 * 24 = 86.4 billion values – almost certainly too many to fit into RAM, and thus rather unwieldy to work with. This begs the question “Is it possible to compute quantiles without storing every observation?”

Visualizing Latency Part 4: Official D3 Latency Heatmap Page

Fri, 12 Jan 2018 00:00:00 +0000

This post is part 4/4 of my Visualizing Latency series.

Allow me to wrap up my visualizing latency post series by noting that my official D3 latency heatmap repository is at https://github.com/sengelha/d3-latency-heatmap/. Monitor this repository for future developments to the D3 latency heatmap chart.

Visualizing Latency Part 3: Rendering Event Data

Fri, 01 Dec 2017 00:00:00 +0000

This post is part 3/4 of my Visualizing Latency series.

Now that I have introduced the D3 latency heatmap chart component and explained what binning is, I can discuss the primary use case of the chart: rendering event data.

What is event data

First, I must explain what I mean by event data. For a fuller treatment, please read Analytics For Hackers: How To Think About Event Data, but allow me to summarize: Event data describes actions performed by entities. It has three key pieces of information: action, timestamp, and state. It is typically rich, denormalized, nested, schemaless, append-only, and frequently extremely large. Some examples of event data include system log records, financial market trades, crime records, or user activities within an application.

Visualizing Latency Part 2: What is Binning?

Thu, 16 Nov 2017 00:00:00 +0000

This post is part 2/4 of my Visualizing Latency series.

As mentioned in Brendan Gregg’s Latency Heat Maps page, a latency heat map is a visualization where each column of data is a histogram of the observations for that time interval. Using Brendan Gregg’s visualization:

As with histograms, the key decision that needs to be made when using a latency heat map is how to bin the data. Binning is the process of dividing the entire range of values into a series of intervals and then counting how many values fall into each interval. That said, there is no “best” number of bins, and different bin sizes can reveal different features of the data. Ultimately, the number of bins depends on the distribution of your data set as well as the size of the rendering area.

Visualizing Latency Part 1: Introduction

Wed, 15 Nov 2017 00:00:00 +0000

This post is part 1/4 of my Visualizing Latency series.

A latency heatmap is a particularly useful tool for visualizing latency. For a great treatment of latency heatmaps, please read Brendan Gregg’s Latency Heat Maps page and the ACM Queue article Visualizing System Latency.

On the right, you can see a latency heatmap generated from a job queueing system which shows a number of interesting properties, not least of which is that the system appears to be getting slower over time.

Data-Driven Code Generation of Unit Tests Part 5: Closing Thoughts

Wed, 05 Jul 2017 19:16:22 +0000

This post is part 5/5 of my Data-Driven Code Generation of Unit Tests series.

In the previous posts in this series, I walked through the idea of performing data-driven code generation for unit tests, as well as how I implemented it in three different programming languages and build systems. This post contains some final thoughts about the effort.

Was it worth it?

Almost certainly. Although it required substantial up-front effort to set up the unit test generators, this approach found numerous, previously-undetected bugs both within my implementation of the calculation library as well as with legacy implementations. It is straightforward to write code generators that test all possible combinations of parameters to the calculations, ensuring that the resulting code coverage is excellent. Adding tests for a new calculation is as straightforward as adding a line to a single file.

Data-Driven Code Generation of Unit Tests Part 4: C#, MSBuild, T4, MS Unit Test

Wed, 05 Jul 2017 16:56:13 +0000

This post is part 4/5 of my Data-Driven Code Generation of Unit Tests series.

This blog post explains how I used C#, MSBuild, T4 Text Templates, and the Microsoft Unit Test Framework for Managed Code to perform data-driven code generation of unit tests for a financial performance analytics library. If you haven’t read it already, I recommend starting with Part 1: Background.

As mentioned in Part 2: C++, CMake, Jinja2, Boost, all performance analytics metadata is stored in a single file called metadata.csv. This file drives all code generation and is what helps ensure inter-platform consistency.

Data-Driven Code Generation of Unit Tests Part 3: Java, Maven, StringTemplate, JUnit

Mon, 03 Jul 2017 00:00:00 +0000

This post is part 3/5 of my Data-Driven Code Generation of Unit Tests series.

This blog post explains how I used Java, Apache Maven, StringTemplate, and JUnit to perform data-driven code generation of unit tests for a financial performance analytics library. If you haven’t read it already, I recommend starting with Part 1: Background.

Data-Driven Code Generation of Unit Tests Part 2: C++, CMake, Jinja2, Boost

Fri, 30 Jun 2017 20:22:36 +0000

This post is part 2/5 of my Data-Driven Code Generation of Unit Tests series.

This blog post explains how I used CMake, Jinja2, and the Boost Unit Test framework to perform data-driven code generation of unit tests for a financial performance analytics library. If you haven’t read it already, I recommend starting with Part 1: Background.

All performance analytics metadata is stored in a single metadata file called metadata.csv. This file contains the complete list of calculations, and for each calculation, its settings (i.e. how it differs from other calculations), including properties like:

Data-Driven Code Generation of Unit Tests Part 1: Background

Fri, 30 Jun 2017 16:42:15 +0000

This post is part 1/5 of my Data-Driven Code Generation of Unit Tests series.

At Morningstar, I created a multi-language, cross-platform performance analytics library which implements both online and offline implementations of a number of common financial analytics such as Alpha, Beta, R-Squared, Sharpe Ratio, Sortino Ratio, and Treynor Ratio (more on this library later). The library relies almost exclusively on a comprehensive suite of automated unit tests to validate its correctness. I quickly found that maintaining a nearly-identical battery of unit tests in three different programming languages was a chore, and I had a hunch that I could use a common technique to deal with this problem: code generation.

Future (circa 2010) Parallel Programming Models

Sat, 26 Dec 2015 00:00:00 +0000

Joe Duffy regularly posts amazing material which is well ahead of our time, such as his current blog post series about Midori.

I’d like to call out this particular assertion made by him way back in 2010:

[D]evelopers must move towards single-threaded programming models connected through message passing, optionally with provably race-free fine-grained parallelism inside of those single-threaded worlds.

Add “async/await everywhere” and you can sign me up!

Threading is a Poor Concurrency Programming Model

Wed, 03 Jun 2015 00:00:00 +0000

Here’s why I try to avoid thread-based programming models for expressing concurrency:

The opponents of thread-based systems line up several drawbacks. For Ousterhout, who probably published the most well-known rant against threads [Ous96], the extreme difficulty of developing correct concurrent code–even for programming experts–is the most harmful trait of threads. As soon as a multi-threaded system shares a single state between multiple threads, coordination and synchronization becomes an imperative. Coordination and synchronization requires locking primitives, which in turn brings along additional issues. Erroneous locking introduces deadlocks or livelocks, and threatens the liveness of the application. Choosing the right locking granularity is also source of trouble. Too coarse locks slow down concurrent code and lead to degraded sequential execution. By contrast, too fine locks increase the danger of deadlocks/livelocks and increase locking overhead. Concurrent components based on threads and locks are not composable. Given two different components that are thread-safe, a composition of them is not thread-safe per se. For instance, placing circular dependencies between multi-threaded components unknowingly can introduce severe deadlocks.

Exploring the .NET CoreFX Part 17: Videotaped API Review

Wed, 04 Mar 2015 00:00:00 +0000

This is part 17/17 of my Exploring the .NET CoreFX series.

Microsoft’s .NET Core team has posted a videotaped API review session where they show how they review API enhancement suggestions. I thought the video was quite educational.

Exploring the .NET CoreFX Part 16: Platform-Specific Builds Using Compile-Time Polymorphism

Sun, 01 Mar 2015 00:00:00 +0000

This is part 16/17 of my Exploring the .NET CoreFX series.

While .NET has historically been limited to Windows machines, Mono notwithstanding, the introduction of the cross-platform .NET Core runtime has introduced the possibility of running .NET Core applications on Unix machines. With this possibility, developers may have the need of writing platform-specific code.

One way to write platform-specific code is:

Define a conceptual base class which will have an identical name and methods across all platforms. This does not need to be a C# interface, as we will be using compile-time rather than run-time polymorphism.
Provide an implementation of this class for each target platform.
Use build-time conditions to include the platform-specific class based on target compilation platform.

An example from the .NET Core is the System.Console.ConsolePal class from the System.Console library. The library includes two implementations of this class:

Exploring the .NET CoreFX Part 15: Using Non-Generic Factory Classes to Enable Type Inference

Wed, 18 Feb 2015 00:00:00 +0000

This is part 15/17 of my Exploring the .NET CoreFX series.

While C# supports type inference for generic methods, it does not support type inference for constructors. In other words, while this code works:

public class FooFactory
{
    public static Foo<T> Create<T>(T value)
    {
        return new Foo<T>(value);
    }
}

var myObj = FooFactory.Create(212);

This code does not:

public class Foo<T>
{
    private readonly T field;
    public Foo(T value) { field = value; }
}

var obj = new Foo(212); // DOES NOT WORK

For more background on why this is, see this StackOverflow post.

Because of this, every generic type in System.Collections.Immutable includes a factory class of the same name which makes construction more convenient. For example:

Exploring the .NET CoreFX Part 14: Inside Immutable Collections

Fri, 13 Feb 2015 00:00:00 +0000

This is part 14/17 of my Exploring the .NET CoreFX series.

Back in 2013, Immo Landwerth and Andrew Arnott recorded a Going Deep video called Inside Immutable Collections which describes how and why System.Collections.Immutable is built the way it is. It’s great background material to understand System.Collections.Immutable.

Exploring the .NET CoreFX Part 13: ImmutableList is an AVL Tree

Tue, 13 Jan 2015 00:00:00 +0000

This is part 13/17 of my Exploring the .NET CoreFX series.

Most implementations of IList, including System.Collections.Generic.List, are dynamic arrays. System.Collections.Immutable.ImmutableList is different – it is an AVL tree. This results in significantly different performance characteristics:

	`List`	`ImmutableList`
Indexing	O(1)	O(log n)
Append	O(1) average, O(n) worst-case	O(log n)
Insert at arbitrary index	O(n)	O(log n)
Remove	O(n)	O(log n)
Memory layout	Contiguous for value types	Non-contiguous

The data structure behind ImmutableList was likely chosen so that modifications to the list are non-destructive and require minimal data copying.

Exploring the .NET CoreFX Part 12: Aggressive Inlining

Mon, 05 Jan 2015 00:00:00 +0000

This is part 12/17 of my Exploring the .NET CoreFX series.

In C++, the inline keyword allows a developer to provide a hint to the compiler that a particular method should be inlined. C# has the identical ability but uses an attribute instead:

internal class SecurePooledObject<T>
{
    ....

    [MethodImpl(MethodImplOptions.AggressiveInlining)]
    internal bool IsOwned<TCaller>(ref TCaller caller)
        where TCaller : struct, ISecurePooledObjectUser
    {
        return caller.PoolUserId == _owner;
    }
}

In System.Collections.Immutable, this attribute is used highly selectively – only once, in fact.

Recommendations

In rare cases, consider using MethodImpl(MethodImplOptions.AggressiveInlining) to suggest to the .NET runtime that a particular method should be inlined.

Win32 LD_PRELOAD

Sat, 20 Dec 2014 00:00:00 +0000

Many versions of UNIX have a useful concept built in to their dynamic linker: by setting an environment variable (usually LD_PRELOAD), a user can force a shared library to load before all others. While I wasn’t able to figure out how to exactly replicate this functionality on Windows, I found a snippit on page 794 of Programming Applications for Microsoft Windows: Fourth Edition, by Jeffrey Richter, on how to come close. On this page, Mr. Richter explains a way to force an executable to load a dynamic library before it begins executing code. While this is only one of several ways he suggests to achieve this goal, I was drawn to this one because it seemed the most flexible.

Exploring the .NET CoreFX Part 11: Code Contracts

Wed, 17 Dec 2014 00:00:00 +0000

This is part 11/17 of my Exploring the .NET CoreFX series.

In 2008, Microsoft Research published Code Contracts, which provide a language-agnostic way to express coding assumptions in .NET programs. The assumptions take the form of pre-conditions, post-conditions, and object invariants.

Here is a simple example of code which uses Code Contracts:

using System.Diagnostics.Contracts;

public class StringUtils
{
    internal static string Append(string s1, string s2)
    {
        Contract.Requires(s1 != null);
        Contract.Requires(s2 != null);
        Contract.Ensures(Contract.Result<string>() != null);
        return s1 + s2;
    }
}

Code Contracts assertions are not limited to runtime enforcement. They may instead be enforced by compile-time static analysis. For example, it is very simple to annotate methods with Code Contracts, set up a continuous integration (CI) server to perform static analysis, and fail the build if there are any failed assertions. This gives us the best of both worlds: a guarantee our code enforces our assumptions with essentially zero runtime penalty.

Exploring the .NET CoreFX Part 10: Performance Tuning Enumeration

Tue, 02 Dec 2014 00:00:00 +0000

This is part 10/17 of my Exploring the .NET CoreFX series.

The .NET Core’s System.Collections.Immutable.ImmutableArray provides two enumerators. The first has been highly tuned for speed, and the second is a fallback for compatibility when it is required.

The high-performance enumerator uses the following performance optimizations:

The enumerator is a struct, rather than a class, so that it is stack-allocated rather than heap-allocated.
The enumerator does not implement IEnumerator or IEnumerator, as this would require it to implement IDisposable. By not implementing IDisposable the iterator will inline during foreach loops..
The enumerator does not use range checks in Enumerator.Current; it requires on .NET’s array range checks to throw an exception instead.

The high-performance enumerator is called ImmutableArray.Enumerator, which is returned by ImmutableArray.GetEnumerator():

Exploring the .NET CoreFX Part 9: Immutable Collections and the Builder Pattern

Mon, 01 Dec 2014 00:00:00 +0000

This is part 9/17 of my Exploring the .NET CoreFX series.

Using the builder pattern to allow for easier construction of immutable objects is well-known.

The .NET Core’s immutable collections assembly, System.Collections.Immutable, also uses the builder pattern, but for a slightly different reason: to improve the performance of making many changes to the collection. This is possible because, unlike the immutable collection itself, the builder pattern does not need to maintain the immutable collection’s invariants after each modification. The builder pattern merely needs to reestablish the invariants of the immutable collection upon the publishing of the results.

Exploring the .NET CoreFX Part 8: NullReferenceException Performance Tricks

Fri, 28 Nov 2014 00:00:00 +0000

This is part 8/17 of my Exploring the .NET CoreFX series.

The .NET Core’s System.Collections.Immutable.ImmutableArray class implements an immutable wrapper around a normal C# managed array. This looks something like:

public struct ImmutableArray<T>
{
    internal T[] array;
    ...
}

ImmutableArray.array is lazy-initialized.

Within the ImmutableArray class, there are a number of methods which have the precondition that ImmutableArray.array must be initialized. These preconditions must be checked before the method begins processing to make sure we handle invalid states correctly.

Exploring the .NET CoreFX Part 7: Reference Versus Structural Equality

Thu, 27 Nov 2014 00:00:00 +0000

This is part 7/17 of my Exploring the .NET CoreFX series.

In the previous post, I referenced EqualityComparer.Default. If T does not implement IEquatable, EqualityComparer.Default will use the framework-defined Object.Equals(), which implements reference equality.

However, many times you want to compare two types for structural equality (i.e. identical content) rather than reference equality (i.e. two references point to the same instance of the class). The interface IStructuralEquatable was defined to allow a class to explicitly implement structural, rather than reference equality. Related classes include IStructuralComparable and StructuralComparisons.

Exploring the .NET CoreFX Part 6: Use IEquatable for Higher-Performance Equals()

Mon, 24 Nov 2014 00:00:00 +0000

This is part 6/17 of my Exploring the .NET CoreFX series.

Let’s say you are writing a custom IList which contains the following code:

public class MyList<T> : IList<T>
{
    private T[] array;

    ...

    public int IndexOf(T item)
    {
        for (int i = 0; i != array.Length; ++i) {
            if (this.array[i].Equals(item)) {
                return i;
            }
        }

        return -1;
    }
}

The above code uses T“s implementation of Object.Equals(), which is defined as:

public virtual bool Equals(Object obj);

If T is a value type, it will be automatically boxed by the compiler, which has a slight performance cost. However, if you knew that T implemented IEquatable, then you could avoid the boxing entirely. For example, this code would be slightly better performing than the above for value types:

Exploring the .NET CoreFX Part 5: Keep Indexers Trivial to Allow JIT Optimization

Fri, 21 Nov 2014 00:00:00 +0000

This is part 5/17 of my Exploring the .NET CoreFX series.

This is a simple recommendation based on observations from System.Collections.Immutable.

Recommendations

Keep the implementation of an indexer as trivial as possible to allow the JIT optimization of removing array bounds checking to work. For example, don’t check if a member variable is null; just use it and allow the NullReferenceException to happen naturally. In other words, use:

public T this[int index]
{
    get
    {
        return this.array[index];
    }
}

not:

Exploring the .NET CoreFX Part 4: The Requires Convenience Class

Thu, 20 Nov 2014 00:00:00 +0000

This is part 4/17 of my Exploring the .NET CoreFX series.

The System.Collections.Immutable project in the .NET CoreFX includes a convenience class called Requires, which looks like:

internal static class Requires
{
    [DebuggerStepThrough]
    public static T NotNull([ValidatedNotNull]T value, string parameterName)
        where T : class // ensures value-types aren't passed to a null checking method
    {
        if (value == null)
        {
            throw new ArgumentNullException(parameterName);
        }

        return value;
    }

    ...
}

This allows other methods to write code like:

public void Foo(string bar)
{
    Requires.NotNull(bar);

    ...
}

Exploring the .NET CoreFX Part 3: Making Methods Debugger-Friendly

Wed, 19 Nov 2014 00:00:00 +0000

This is part 3/17 of my Exploring the .NET CoreFX series.

System.Collections.Immutable uses a number of attributes to make it more debugger-friendly. Here are the key attributes:

DebuggerStepThrough

Occasionally a method is so simple that it doesn’t make sense to have the debugger step into it. The System.Diagnostics.DebuggerStepThroughAttribute instructs the debugger to step through the code instead of stepping into the code.

Here is an example from System.Collections.Immutable:

internal static class Requires
{
    [DebuggerStepThrough]
    public static void Range(bool condition, string parameterName, string message = null)
    {
        if (!condition)
        {
            FailRange(parameterName, message);
        }
    }
}

DebuggerBrowsable

The System.Diagnostics.DebuggerBrowsableAttribute determines if and how a member is displayed in the debugger variable windows.

Exploring the .NET CoreFX Part 2: Cache ThreadLocal Variables in Locals

Tue, 18 Nov 2014 00:00:00 +0000

This is part 2/17 of my Exploring the .NET CoreFX series.

Thread-local storage allows you to mark a global or static variable as local to a thread. In Win32, thread-local storage is provided by the functions TlsAlloc, TlsGetValue, TlsSetValue, and TlsFree. Similarly, C# provides System.ThreadStaticAttribute and System.Threading.ThreadLocal.

Unfortunately, thread-local storage comes at a cost. Reading or writing a thread-local variable is far more expensive than reading or writing a local variable. System.Collections.Immutable uses a trick or two to help ameliorate this expense. For example, System.Collections.Immutable caches thread-local variables in local variables in a method to avoid unnecessary TLS hits on repeated access. Here’s some sample code which implements this:

Exploring the .NET CoreFX Part 1: Annotate Pure Methods With PureAttribute

Mon, 17 Nov 2014 00:00:00 +0000

This is part 1/17 of my Exploring the .NET CoreFX series.

A pure method is a method that does not make any visible state changes.

John Carmack, in his article In-Depth: Functional Programming in C++, notes many advantages of pure functions:

Pure functions have a lot of nice properties.

Thread safety. A pure function with value parameters is completely thread safe. With reference or pointer parameters, even if they are const, you do need to be aware of the danger that another thread doing non-pure operations might mutate or free the data, but it is still one of the most powerful tools for writing safe multithreaded code.

The Power of Central Banks

Wed, 14 May 2014 00:00:00 +0000

In Europe, the transformation in the sovereign debt market since Mario Draghi’s “whatever it takes” promise in July 2012 has been remarkable. Ireland, which just exited an IMF bailout program in December, can now issue 5-year debt at yields well below 5-year Treasuries.

– PIMCO Secular Outlook, May 2014 – The New Neutral

Put all Maven dependency versions in properties

Fri, 02 May 2014 00:00:00 +0000

Because security vulnerabilities in third-party software are so common, it is important to manage the versions of your dependencies and to be prepared to upgrade them quickly. For Maven projects, I recommend putting the versions of all your dependencies in the properties section of your pom.xml. Furthermore, for libraries which require multiple dependency entries in pom.xml, this allows you to ensure you use the same version for all of those dependencies.

For example:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <properties>
        <spring.version>3.2.6.RELEASE</spring.version>
        <jackson.version>2.2.3</jackson.version>
    </properties>
    <dependencies>
        <!-- spring -->
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-core</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-beans</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-context</artifactId>
            <version>${spring.version}</version>
        </dependency>
        <!-- Jackson -->
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-core</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-annotations</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
    </dependencies>
</project>

Publishing a Java Servlet to Azure Website Using Maven

Thu, 01 May 2014 00:00:00 +0000

This blog post shows how to publish a Java Servlet, encapsulated in a WAR file, to an Azure Web Site using FTP via Maven.

The first step will be to generate a simple servlet using Maven:

mvn archetype:generate -DgroupId=com.example -DartifactId=hello-world -DarchetypeArtifactId=maven-archetype-webapp

This creates a simple, Hello World application in the hello-world directory. We can verify it works by running it in a local servlet container using the instructions found in Supporting mvn jetty:run in Maven applications. Add the Jetty maven plugin to pom.xml under the build section:

Supporting mvn jetty:run in Maven applications

Wed, 30 Apr 2014 00:00:00 +0000

When I’m writing a Java servlet using Maven, I find it convenient to be able to run the Java servlet in a local servlet container for testing purposes. This is very easy to do using the Jetty Maven plugin.

To add the Jetty Maven plugin to your project, modify pom.xml as follows:

<build>
    <plugins>
        <plugin>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-maven-plugin</artifactId>
            <version>9.2.0.M0</version>
        </plugin>
        ...
    </plugins>
</build>

You can then run your project in Jetty using the following command:

Work Email Statistics Through 2013-12-31

Mon, 28 Apr 2014 00:00:00 +0000

In a previous post, I noted that I have a copy of every single email I’ve sent or received at work since mid-2003. Here is the updated chart of the number of work emails I send and receive per year through December 31, 2013.

Here’s the same data charted by month:

This data was calculated by a simple Outlook PST parsing program I wrote called pststats.

Using Handbrake to Encode Blu-Rays for the Roku

Sun, 27 Apr 2014 00:00:00 +0000

This post is a sibling of my earlier post Using Handbrake to Encode DVDs for the Roku and contains the settings I use for Blu-Ray movies. The instructions apply to Handbrake 0.9.9.

Once again, start with the High Profile preset and then make the following changes:

In Output Settings:
- Set Container = Mp4
- Ensure Large File Size is checked
- Ensure Web Optimized is checked
- Ensure iPod 5G Support is unchecked
In Picture:
- Set Anamorphic = Strict (all Blu-Rays I’ve seen are 1920x1080)
- Set Cropping = Automatic.
In Filters:
- Set Detelecine = Off
- Set Decomb = Fast
- Set Deinterlace = Off
- Set Denoise = Off
- Set Deblock = Off
- Ensure Grayscale is unchecked
In Video:
- Set Video Codec = H.264 (x264)
- Set Framerate (FPS) = Same as source
- Set Constant Framerate
- Set Quality = Constant Quality of RF:20
- Set Optimise Video = Use Advanced Tab instead
In Audio:
- Create two audio tracks. The first audio track will have:
  - Source = AC3 5.1 (which is what you should have ripped from your DVD)
  - Codec = AAC (faac)
  - Bitrate = 160
  - Samplerate = Auto
  - Mixdown = Stereo
  - DRC = 1.5
  - Gain = 0
- The second audio track will have:
  - Source = AC3 5.1
  - Codec = AC3 Passthru
In Subtitles, keep everything empty. I do not handle subtitles.
In Chapters:
- Ensure Create chapter markers is checked
In Advanced:
- In Encoding:
  - Set Reference Frames = 3 (Default)
  - Set Maximum B-Frames = 3 (Default)
  - Ensure CABAC is checked
  - Ensure 8×8 Transform is checked
  - Ensure Weighted P-Frames is checked
  - Set Pyramidal B-Frames = Normal (Default)
- In Analysis:
  - Set Adaptive B-Frames = Optimal
  - Set Adaptive Direct Mode = Automatic
  - Set Motion Est Method = Hexagon (Default)
  - Set Subpixel Motion Est = 7: RD in all frames
  - Set Motion Est Range to 16
  - Set Partition Type = Most (Default)
  - Set Trellis = Encode Only (Default)
- In Psychovisual:
  - Set Adaptive Quant Strength = Middle
  - Set Rate Distortion = Middle
  - Set Psychovisual Trellis = Leftmost Setting
  - Ensure No DCT-Decimate is unchecked
  - Set Deblocking = 0 (Default), 0 (Default)
- In x264 Encoder Options, set it to b-adapt=2:rc-lookahead=50:direct=auto

Using Handbrake to Encode DVDs for the Roku

Mon, 02 Dec 2013 00:00:00 +0000

Update 2014-04-27 10:00 PM: See also Using Handbrake to Encode Blu-Rays for the Roku

In a previous post, I noted that I use ffmpeg to convert DVDs to MP4s which will play on my Roku. I recently encountered two problems with this approach. First, the resulting MP4s only include a stereo audio track, so are highly sub-optimal for surround sound systems. Second, for a yet-undetermined reason, Plex (my current preferred media client) will transcode the videos rather than play them directly.

Instrumenting a Java Web Application with JMX

Mon, 25 Nov 2013 00:00:00 +0000

Java Management Extensions (JMX) is a technology for managing and monitoring Java applications. JMX instructs application developers how to instrument their applications to expose metrics and management mechanisms, and instructs operations teams how to collect metrics and manage Java applications. JMX is ubiquitous: there are a large number of off-the-shelf commercial and open source tools which speak JMX.

This article is intended for application developers to introduce JMX, to teach them how to create some simple metrics via JMX, and to encourage them to thoroughly instrument their applications with JMX.; This article uses only J2SE 5 and the Java Servlet specification. There may be better, more efficient, or more appropriate ways to add JMX counters to applications using other frameworks. For example, Spring developers should read http://static.springsource.org/spring/docs/2.0.x/reference/jmx.html.

Clean Code is Not Threadsafe Code?

Sat, 05 Oct 2013 00:00:00 +0000

I am currently reading Clean Code: A Handbook of Agile Software Craftsmanship by Robert C. Martin and I’ve noticed a pattern.

In Listing 2-1, Martin recommends refactoring this code:

private void printGuessStatistics(char candidate, int count)
{
    String number;
    String verb;
    String pluralModifier;
    if (count == 0) {
        number = "no";
        verb = "are";
        pluralModifier = "s";
    } else if (count == 1) {
        number = "1";
        verb = "is";
        pluralModifier = "";
    } else {
        number = Integer.toString(count);
        verb = "are";
        pluralModifier = "s";
    }
    String guessMessage = String.format(
        "There %s %s %s%s", verb, number, candidate, pluralModifier
    );
    print(guessMessage);
}

into this:

Ffmpeg encoding for the Roku

Sun, 07 Jul 2013 00:00:00 +0000

Update 2013-12-02: I no longer use this method. See this post for my new method.

I have three (someday to be four) Roku players at home. I primarily use them to stream Netflix and Amazon Instant Video and to stream MP4s from a home server using Roksbox.

My typical workflow is to rip DVDs to VOBs using Dvdfab, copy the VOBs to my home server, and then convert them in bulk from VOB to MP4 using ffmpeg.

Creating a Virtual Java RPM

Wed, 29 May 2013 00:00:00 +0000

Some RPMs (e.g. jpackage.org’s tomcat7-7.0.39-1.jpp6.noarch.rpm) express their dependency upon Java by requiring a RPM that provides capability java (as opposed to, for example, depending on the existence of a file /usr/bin/java). On CentOS, this capability is normally provided by the java-*-openjdk RPM. Therefore, if you execute # yum install tomcat7 on a clean install of CentOS, yum will install OpenJDK in addition to Tomcat 7.

Some people prefer to run the Oracle JRE/JDK instead of OpenJDK. Oracle provides RPMs named jre-version-linux-x64.rpm and jdk-version-linux-x64.rpm to make installing them easier. Unfortunately, these RPMs do not provide the capability java. This means that if you already have the Oracle JRE installed, and you install a RPM which requires the capability java, the OpenJDK will be unnecessarily installed (and might even become the default!).

Pronounciation of GIF

Tue, 21 May 2013 00:00:00 +0000

The Oxford English Dictionary accepts both pronunciations [of the word GIF]. They are wrong. It is a soft ‘G,’ pronounced ‘jif.’ End of story.

– Steve Wilhite, creator of the GIF file format

Booting an Acer Aspire easyStore H340 from External USB

Sun, 19 May 2013 00:00:00 +0000

A couple of years ago I bought a Acer Aspire easyStore H340 to use as a home NAS device. While the H340 came with Windows Server, I decided to replace it with Debian Linux running on an external USB stick. Besides using it for basic file storage, I use Nginx to stream MP4s to Roku devices throughout my home using Roksbox; I use forked-daapd to stream FLACs to iTunes (albeit with limited success – foobar2000 pointing to the CIFS file share works better); and I use CrashPlan to back up my important data (e.g. pictures) to the Internet.

Varnish and Graphite

Mon, 29 Apr 2013 23:50:50 +0000

At work we have a reasonably busy Varnish cluster. We hooked it up to Graphite to allow near real-time visualization of how busy it is. Here’s a sample of what part of our Graphite dashboard looks like when you look over the past four months:

Effects of Windows 2008R2

Mon, 29 Apr 2013 12:49:40 +0000

The effects of upgrading a (deliberately unnamed) web application from Windows 2003 to Windows 2008R2:

Passive Indexing Wins Again

Tue, 09 Apr 2013 00:00:00 +0000

I’m a bit of a Boglehead, and the data appears to back me up. From http://www.cbsnews.com/8301-505123_162-57578430/active-managers-lost-again-in-2012/

In 2012, 63 percent of large-cap funds, 80 percent of mid-cap funds and 67 percent of small-cap funds underperformed [relative to their benchmark]. The only asset class to see the majority of active funds outperform was large-cap growth, with 54 percent beating their benchmarks. The worst performance came in mid-cap growth funds where 87 percent failed to outperform.

Scaling Varnish

Mon, 08 Apr 2013 03:52:52 +0000

As seen on varnish-misc@varnish-cache.org:

I read one web testimony of a person who used Varnish to scale a site up to almost 200 million page views a month.

I’d like to find someone who has that level of expertise.

At work we have a Varnish cluster which, during the month of March, served over 17 billion requests for an average of 6,400 requests/second.

This cluster contains two physical machines with 24 cores and 192GB of RAM apiece. These machines have tons of spare capacity; their load average peaked at approximately 4. We’ve even considered virtualizing the cluster.

Technology Koans

Mon, 08 Apr 2013 03:23:38 +0000

I find these amusing and thought-provoking:

Unix File Extension

Fri, 05 Apr 2013 00:00:00 +0000

This useful little snippet can be used to find the unique list of extensions for all files in a directory and all its subdirectories. I can never remember the precise awk incantation for this:

% find . -type f | awk -F. '{print $NF}' | tr '[:upper:]' '[:lower:]' | sort -u

MySQL is (Reportedly) Still Terrible

Sun, 10 Feb 2013 00:00:00 +0000

Many years ago, when I actually paid attention to such things, I rejected MySQL in favor of PostgreSQL because of the former’s terrible defaults (e.g. no ACID guarantees on the default storage engine at the time, MyISAM) and missing features (in 3.23, no subqueries, stored procedures, views, etc.). The only time I regretted it was when dealing with other software that required MySQL and would not work with PostgreSQL.

Looks like not much has changed.

Python Multiprocessing Module and Closures

Wed, 16 Jan 2013 00:00:00 +0000

At work, I wrote a Python script which uses the multiprocessing module to process many servers in parallel. The code looks something like:

def processServer(server):
    # Do work...

numParallelTasks = ...
servers = [...]
pool = multiprocessing.Pool(processes=numParallelTasks)
results = pool.map(processServer, servers)

I wanted to pass some extra state to processServer without using a global variable. My first attempt was to use a closure, so I wrote the following:

def processServer(extraState):
    def processServerWorker(server):
        # Do work, using extraState as needed
    return processServerWorker

numParallelTasks = ...
servers = [...]
extraState = ...
pool = multiprocessing.Pool(processes=numParallelTasks)
results = pool.map(processServer(extraState), servers)

This failed with the following error:

Work Email Statistics

Wed, 02 Jan 2013 00:00:00 +0000

I have a copy of every single email I’ve sent or received at work since mid-2003. Here is a chart of the number of work emails I send and receive per year.

Loss of Context Leads to Absurdity

Wed, 08 Aug 2012 00:00:00 +0000

Seen in an email:

A copy of Spongebob seems to be sitting on a file server at Xxxxxxxx.

Spongebob is the codename for an product.

Longest Interactive Command Yet

Thu, 02 Aug 2012 00:00:00 +0000

This is perhaps the longest command I’ve ever entered on the interactive command line:

[113 sengelha@centosdev-vm]% cd modules/puppet && git init && git add --all
&& git commit -m "Initial Commit" && git remote add origin https://xxxxxxx@x
xxxxxx/scm/PM/puppet.git && git push origin master && cd ../.. && git rm -r
modules/puppet && rm -rf modules/puppet && git submodule add https://xxxxxxx
/scm/PM/puppet.git modules/puppet && git commit -m "Move puppet to submodule
" && git push

It moves a Puppet module to its own, separate git project and sets up a Git submodule reference to the moved module.

Effects of Akamai

Fri, 27 Jul 2012 03:09:53 +0000

The effects of putting a (deliberately unnamed) web service behind Akamai:

Varnish Expiration Thread Lock Contention

Fri, 27 Jul 2012 02:38:11 +0000

I recently helped introduce a Varnish cluster at work to offload a significant number of requests from a heavily-trafficked internal web service. This Varnish cluster serves approximately 10,000 requests/second (with peaks in the 20-30,000 requests/second range) 24x7x365.

While working on this project I ran into a few interesting problems with Varnish. The most interesting one I saw was that Varnish would consume more and more RAM until eventually it would consume all the memory on the box and crash. Using varnishstat, we diagnosed that this was due to unbounded growth of transient storage within the Varnish process.

Reader/Writer Lock Pattern

Thu, 04 Mar 2010 00:00:00 +0000

A reader-writer lock is a lock which will allow multiple concurrent readers but only one writer. A reader-writer lock can be significantly more efficient than a standard mutex if reads on your shared memory far outnumber writes.

Reader-writer locks naturally fit together with caches, as caches are only effective if reads far outnumber writes.

Here is a general pattern for using a reader-writer lock with a cache:

Acquire a reader lock.
Check the cache for the value. If it exists, save the value and go to step 8.
Upgrade the reader lock to a writer lock.
Check the cache for the value. If it exists, save the value and go to step 7.
Calculate the value (expensive, otherwise we wouldn’t cache it)
Insert the value into the cache.
Release the writer lock.
Release the reader lock.
Return the value.

The reason why we have to check the cache for the value again in step (4) is because of the following possibility (assume step 4 doesn’t exist):

Handling Multiple QueryString Parameters With the Same Key in ASP.NET

Wed, 23 Sep 2009 00:00:00 +0000

When you are processing an HTTP request in ASP.NET you can retrieve the user-provided query string parameters using the HttpRequest.QueryString property. This property is an instance of the NameValueCollection class.

If the user has provided multiple parameters with the same key in the query string, HttpRequest.QueryString[key] will return all the values concatenated together with commas. If you would rather process the values individually, use HttpRequest.QueryString.GetValues(key), which will return an array of all the provided values.

Balloon Tooltips

Thu, 12 Jun 2008 00:00:00 +0000

In the Windows XP login screen, the password text box will warn you with a balloon tooltip if you accidentally turn Caps Lock on. The balloon tooltip appears to be a Windows tooltip common control with the TTS_BALLOON style.

To replicate this functionality, I decided to write a function called ShowMsgBalloon() which, given a control and the various balloon tooltip parameters, creates and shows the balloon tooltip below the control.

Escaping Strings in XPath 1.0

Tue, 03 Jun 2008 00:00:00 +0000

XPath is a language for selecting nodes from an XML document. XPath is used extensively in XSLT and other XML technologies. I also vastly prefer using XPath (e.g. with XPathNavigator) over the XML DOM when manipulating XML in a non-streaming fashion.

In XPath, strings must be delimited by either single or double quotes. Given a quote character used to delimit a string, one can’t represent that same quote character within the string. This means that if you decide to use single quotes to delimit your XPath string, you couldn’t represent the string O’Reilly; use double quotes, and you can’t represent “Hello”.

What I Learned Tonight…

Sat, 05 Apr 2008 00:00:00 +0000

Diagnosing IIS NTLM authentication failures is a real pain, until…
You use Microsoft’s Authentication and Access Control Diagnostics package, which can tell you…
You need to enable HTTP Keep-Alives for NTLM to work.

Representing Date/Times as Strings for Non-Human Consumption

Sat, 08 Mar 2008 00:00:00 +0000

If you ever have the need to represent a date/time (or part of a date/time) as a string for programmatic rather than human consumption (e.g. you are defining a save file format or a network protocol), please use ISO 8601 unless you have a very strong reason not to.

For more information, please read what the W3C has to say about ISO 8601 style date and time formats.

Don't Forget to Reap your Zombies

Fri, 07 Mar 2008 00:00:00 +0000

I recently received a bug report for my quick-and-dirty TCP debugging tool tcpconndbg where it was creating a large number of zombie processes. The person who filed the bug, Peter Viskup, was even kind enough to send a patch. While this is old news to anyone with extensive Unix programming experience, always remember the following:

If you create a child process using fork(), you must either:

Explicitly retrieve the child process’s exit code using one of the wait() functions (e.g. waitpid())
Tell the system that you aren’t interested in the child process’s exit code by using either:
1. sigaction() with the SA_NOCLDWAIT parameter (preferred)
2. signal(SIGCHILD, SIG_IGN); (for systems which do not support sigaction())

As I fixed this bug, I realized I hadn’t looked at tcpconndbg in 5 years. My how programming style changes…

Geometric Annual Return in SQL

Wed, 30 Jan 2008 00:00:00 +0000

Here is some quick-and-dirty SQL to calculate an geometric annual return (as a percent) from a column of monthly returns (in percents).

/* Convert the annualized number back to a percent */
SELECT (T3.AnnHPR - 1) * 100 AS GeomAnnRet
FROM
  (
  /* Annualize the holding period return */
  SELECT POWER(T2.HPR, 12.0 / T2.NumReturns) AS AnnHPR
  FROM
    (
    /* Calculate the holding period return over the time
       period.

       POWER(10, SUM(LOG10(n))) is a simulated PRODUCT(n)
       aggregate function.

       The precision of POWER is determined by the precision
        of the first argument, so use a lot of decimals. */
    SELECT POWER(10.0000000000000000,
                 SUM(LOG10(T.MonthReturn))) AS HPR,
           COUNT(*) AS NumReturns
    FROM
      (
      /* Convert all percent returns to multipliers (1% ->
         1.01) */
      SELECT 1 + MonthPctReturn / 100 AS MonthReturn
      FROM ...
      ) AS T
    ) AS T2
  ) AS T3

Update 2008-01-30 10:52PM: Here’s the equivalent “one-liner”:

STL Objects and Win32 Module Boundaries

Fri, 04 Jan 2008 00:00:00 +0000

Let’s say you have the following function:

void AppendChar(std::string& s, char ch)
{
    s += ch;
}

What happens if this function is exported as an ordinal function from a DLL (not an inlined piece of code inside a header) and you call it from an EXE?

It works most of the time. When it doesn’t, it corrupts your heap and causes a spectacular mess.

In Windows you must free memory with the same allocator that allocated it. However, your EXE may not share the same allocator as the DLL. Perhaps the two modules are linked against different versions of libc, or perhaps one of the modules is using a static version of libc. If your EXE and DLL do not share an allocator and if AppendChar() resizes the string s, you will almost certainly cause a heap corruption.

Custom-Drawn Win32 Tooltips

Wed, 29 Aug 2007 00:00:00 +0000

Like many common controls, the tooltip control supports custom drawing for maximum flexibility. This is a quick tutorial on how to use the tooltip custom draw facility.

First, start with the following scratch program (which is a slightly modified version of Raymond Chen’s scratch program):

#define STRICT
#include <windows.h>
#include <windowsx.h>
#include <commctrl.h>
#include <tchar.h>

#define WND_CLASS_NAME TEXT("Scratch")

HINSTANCE g_hinst;

BOOL OnCreate(HWND hwnd, LPCREATESTRUCT lpcs)
{
    return TRUE;
}

void OnDestroy(HWND hwnd)
{
    PostQuitMessage(0);
}

LRESULT CALLBACK WndProc(HWND hwnd, UINT uiMsg, WPARAM wParam,
                         LPARAM lParam)
{
    switch (uiMsg)
    {
    HANDLE_MSG(hwnd, WM_CREATE, OnCreate);
    HANDLE_MSG(hwnd, WM_DESTROY, OnDestroy);
    }

    return DefWindowProc(hwnd, uiMsg, wParam, lParam);
}

BOOL RegisterWindowClass()
{
    WNDCLASS wc;
    ATOM atom;

    wc.style = 0;
    wc.lpfnWndProc = WndProc;
    wc.cbClsExtra = 0;
    wc.cbWndExtra = 0;
    wc.hInstance = g_hinst;
    wc.hIcon = NULL;
    wc.hCursor = LoadCursor(NULL, IDC_ARROW);
    wc.hbrBackground = (HBRUSH)(COLOR_WINDOW + 1);
    wc.lpszMenuName = NULL;
    wc.lpszClassName = WND_CLASS_NAME;

    atom = RegisterClass(&wc);
    return (atom != 0);
}

int WINAPI _tWinMain(HINSTANCE hinst, HINSTANCE hinstPrev,
                     LPTSTR lpCmdLine, int nCmdShow)
{
    INITCOMMONCONTROLSEX icc;
    int ret = EXIT_FAILURE;

    g_hinst = hinst;

    // We will need the tooltip common control
    icc.dwSize = sizeof(icc);
    icc.dwICC = ICC_WIN95_CLASSES;
    if (InitCommonControlsEx(&icc))
    {
        if (RegisterWindowClass())
        {
            HWND hwnd = CreateWindow
                (
                WND_CLASS_NAME,
                TEXT("Scratch"),
                WS_OVERLAPPEDWINDOW,
                CW_USEDEFAULT, CW_USEDEFAULT,
                CW_USEDEFAULT, CW_USEDEFAULT,
                NULL,
                NULL,
                hinst,
                );
            if (hwnd != NULL)
            {
                MSG msg;

                (void) ShowWindow(hwnd, nCmdShow);
                while (GetMessage(&msg, NULL, 0, 0)) {
                    TranslateMessage(&msg);
                    DispatchMessage(&msg);
                }

                ret = EXIT_SUCCESS;
            }
        }
    }

    return ret;
}

Next, we’ll add a tooltip to this window. We’re not going to do anything fancy like tooltip multiplexing so we’ll use TTF_SUBCLASS.

Converting C++ Member Functions into Function Objects

Tue, 28 Aug 2007 00:00:00 +0000

Let’s say you have a C++ function that takes a function object as a parameter and calls it:

template <typename _Fn>
void call_functor(_Fn fn)
{
    fn();
}

Now let’s say you want to pass a class’s member function to call_functor() above, as in:

class C
{
    void foo() { std::cout << "foo()\n"; }
};

C c;
call_functor(/\* What do I put here? c.foo and &C::foo don't work \*/);

The STL has a pointer-to-member function adapter called std::mem_fun() which almost gets us there. Unfortunately, it doesn’t quite meet our needs because it requires us to pass a pointer to an instance of C, as in:

Win32: Getting LOGFONT from HFONT

Wed, 22 Aug 2007 00:00:00 +0000

To convert a HFONT to a LOGFONT, use the GDI function GetObject(), as in:

LOGFONT lf;
int ret = GetObject(hfont, sizeof(lf), &lf);
// Be sure to check the return value of GetObject

The code is trivial but the function took me forever to find.

Microsoft’s XmlLite

Thu, 12 Jul 2007 00:00:00 +0000

Microsoft has created a new, lightweight C++ XML processing library called XmlLite. It includes a streaming XML writing class patterned after .NET’s System.Xml.XmlWriter.

This library makes the IXmlWriter in Implementing IXmlWriter Series obsolete for Windows developers.

Generating and Parsing Localized Numbers In Windows

Wed, 11 Jul 2007 00:00:00 +0000

While Windows supports dozens or even hundreds of languages, its localization APIs require quite a bit of getting used to. Below is how I solved some common problems related to formatting and parsing a number for a specific locale.

Formatting a Number for a Locale

The function GetNumberFormat() formats a number for a particular locale. Its simplest usage looks something like:

#define ARRAYSIZE(x) ( sizeof(x) / sizeof(x[0]) )

TCHAR buf[80];
int ret = GetNumberFormat
    (
    LOCALE_USER_DEFAULT, // locale
    0,                   // dwFlags
    TEXT("1234567.89"),  // lpValue
    NULL,                // lpFormat
    buf,                 // lpNumberStr
    ARRAYSIZE(buf)       // cchNumber
    );
ASSERT(ret != 0);

buf now contains the number 1234567.89 formatted for the user’s default locale. For example, for the English-United States locale, buf will contain “1,234,567.89”; for German-Germany, “1.234.567,89”; for Hindi, “12,34,567.89”.

XmlTextWriter Can Produce Invalid XML

Sat, 16 Jun 2007 00:00:00 +0000

XmlTextWriter is .NET’s class for writing XML in a forward-only streaming manner. It is highly efficient and is the preferred way to generate XML in .NET in most circumstances. I find XmlTextWriter so useful I wrote a partial C++ implementation of it in Implenting IXmlWriter Series.

Unfortunately, XmlTextWriter isn’t quite as strict as it could be. It will let slip some invalid XML such as duplicate attributes, invalid Unicode characters in the range 0×0 to 0×20, and invalid element and attribute names. You can read about XmlTextWriter’s limitations in the article Customized XML Writer Creation.

Vista Does Not Virtualize Creation Of Shell Links

Thu, 24 May 2007 00:00:00 +0000

Windows Vista developers beware: Vista does not perform file virtualization on the creation of shell links. Consider the following code:

// Creates a shell link (a.k.a. shortcut) located at swzLinkFile that points to
// szTargetFile with a description of szDescription.
BOOL CreateLink(LPCTSTR szTargetFile, LPCTSTR szDescription, LPCOLESTR swzLinkFile)
{
    BOOL bRet = FALSE;

    IShellLink* psl;
    HRESULT hr = ::CoCreateInstance(CLSID_ShellLink, NULL, CLSCTX_INPROC_SERVER,
                                    IID_IShellLink, (void**) &psl);
    if (SUCCEEDED(hr))
    {
        IPersistFile* ppf;
        hr = psl->QueryInterface(IID_IPersistFile, (void**) &ppf);
        if (SUCCEEDED(hr))
        {
            hr = psl->SetPath(szTargetFile);
            if (SUCCEEDED(hr))
            {
                hr = psl->SetDescription(szDescription);
                if (SUCCEEDED(hr))
                {
                    hr = ppf->Save(swzLinkFile, TRUE);
                    if (SUCCEEDED(hr))
                    {
                        bSuccess = TRUE;
                    }
                }
            }
            ppf->Release();
        }
        psl->Release();
    }
    return bSuccess;
}

// NOTE: Hardcoding C:\WINDOWS and C:\Program Files is a bad practice. Use
// something like ::SHGetFolderPath().
BOOL bSuccess = CreateLink
    (
    _T("C:\\WINDOWS\\SYSTEM32\\SOL.EXE"),
    _T("Shortcut to SOL.EXE"),
    L"C:\\Program Files\\sol.lnk")
    );

One might expect that the creation of the file C:\Program Files\sol.lnk would be silently redirected by Vista using file virtualization and CreateLink() would succeed, but it doesn’t — the call to IPersistFile::Save() returns E_ACCESSDENIED.

Debugging Crashes in Windows Applications: The Null Pointer Dereference

Wed, 25 Apr 2007 00:00:00 +0000

Windows C++ developers remain all too familiar with the standard Windows crash dialog. This post is an attempt to teach developers how to understand the data the crash dialog reports to diagnose difficult issues. A basic understanding of assembly language is assumed; for more background on these topics please read Matt Pietrek’s “Under The Hood” articles in the Microsoft Systems Journal February 1998 and June 1998 issues.

To begin with, let’s write an application that crashes with a null pointer dereference:

Delta Compression Application Programming Interface

Mon, 12 Mar 2007 00:00:00 +0000

Apparently Microsoft has an API for just about everything. Today I read about Microsoft’s Delta Compression Application Programming Interface, an API for creating and applying binary diffs. This API looks ideal for building an application’s incremental online update facility.

Win32 Shell Lightweight Utility API (shlwapi.dll)

Thu, 01 Feb 2007 00:00:00 +0000

The Win32 shell lightweight utility API (shlwapi.dll) is a cornucopia of useful functions. It appears to be Microsoft’s “dumping ground” for functions without a better home. (I believe Microsoft internal DLL ownership also played a part.) Had I known about shlwapi years ago, it would have saved me considerable programming effort.

Particularly useful are SHCreateStreamOnFile, the path family of functions (e.g. PathCombine), and the registry family of functions (e.g. SHRegGetPath).

However, it appears that Microsoft is slowly moving functionality out of shlwapi into other places. For example, many of shlwapi’s string handling functions have better-designed replacements in Microsoft’s safe string library strsafe. Similarly, shlwapi’s SHRegGetValue has been deprecated in favor of advapi32.dll’s RegGetValue.

STL Map Use

Thu, 25 Jan 2007 00:00:00 +0000

What’s wrong with the following code?

template<typename T1, typename T2>
struct my_pair
{
    typedef T1 first_type;
    typedef T2 second_type;

    my_pair() : first(T1()), second(T2()) {}
    my_pair(const T1& v1, const T2& v2) : first(v1), second(v2) {}

    T1 first;
    T2 second;
};

template<typename T1, typename T2>
inline bool operator<
    (
    const my_pair<T1, T2>& x,
    const my_pair<T1, T2>& y
    )
{
    return (x.first < y.first || x.second < y.second);
}

void f()
{
    typedef my_pair<..., ...> key_type;
    typedef ... value_type;
    typedef std::map<key_type, value_type> map_type;

    map_type map;
    // Use map
}

Answer: my_pair cannot be used as a key for a STL map because the operator< violates the rule of strict weak ordering. More specifically, the operator is not antisymmetric. Consider the following:

STL Vector Use

Tue, 23 Jan 2007 00:00:00 +0000

I recently wrote a piece of code that looked something like the following:

static const int NUM_TOTAL_VALUES = ...;
typedef ... T;

// Create vec and reserve NUM_TOTAL_VALUES spaces for later insertion
std::vector<T> vec(NUM_TOTAL_VALUES);

// Insert values into vec
for (int i = 0; i != NUM_TOTAL_VALUES; ++i)
    vec.push_back(...);

// vec should now have NUM_TOTAL_VALUES values in it (but doesn't!)

What’s wrong with this code?

The constructor vector(size_type _Count); does more than just allocate enough space to store _Count items — it also inserts _Count (default constructed) items into the vector. To reserve space without actually inserting values, use reserve():

Managed Wrappers and Hidden Interdependencies

Wed, 10 Jan 2007 00:00:00 +0000

Let’s say you have the following unmanaged code:

#pragma unmanaged

class Stream { ... }; // Conceptual stream class

class StreamWriter
{
public:
    StreamWriter(Stream* pStream) : m_pStream(pStream) {}
    ~StreamWriter() { /* Use m_pStream in some way */ }

    ...

private:
    Stream* m_pStream;
};

void f()
{
    Stream stream;
    StreamWriter streamWriter(&stream);

    // Use streamWriter
    // streamWriter is destroyed
    // stream is destroyed
}

Note that StreamWriter’s destructor uses m_pStream (perhaps by flushing the stream). This means that the order of destruction is important — StreamWriter must be destroyed before its underlying Stream is.

How Return XML From ASPX in ASP.NET 1.1

Mon, 06 Feb 2006 00:00:00 +0000

I’m not sure if this is the “canonical” way to do it but here’s a description of how to write an ASP.NET 1.1 ASPX page which returns a XML document (e.g. when writing a home-brewed web service).

First, create a new Web Form (I will call it WebService.aspx). As we will be progamatically generating the XML in the HTTP response rather than sending the (processed) content of the ASPX file, delete everything from the ASPX file but the @Page directive, so that it looks something like:

XSLT Number Formatting Notes

Thu, 02 Feb 2006 00:00:00 +0000

When using XSLT’s format-number() function to format a decimal, consider using a zero in the least significant place of the decimal part of your format string. This will allow a number with a 0 integer part to display correctly.

For example:

Number	format-number using #,###	format-number using #,##0
12345	12,345	12,345
5	5	5
	No output!

This also applies to decimals:

Number	format-number using #.00	format-number using 0.00
5	5.00	5.00
	.00 (Note no leading 0)	0.00
0.1234	.12 (Note no leading 0)	0.12

C++ Exception Handling

Mon, 30 Jan 2006 00:00:00 +0000

Quick quiz: What is the behavior of the following code:

#include <stdio.h>

int main(int argc, char* argv[])
{
    try
    {
        *((char*) 0) = 1;
    }
    catch (...)
    {
        printf("Caught exception.\n");
    }

    return 0;
}

Surprisingly, it depends! As I would expect, on many operating system / compiler combinations (such as gcc version 3.4.4 20050721 (Red Hat 3.4.4-2) as tested by Keith Garner), the dereference of the null pointer leads to a segmentation fault. However, both Microsoft Visual Studio 6.0 and Visual Studio .NET 2003 print the message Caught exception. and exit gracefully. Strange!

Implementing IXmlWriter Part 14: Supporting Writing To A Stream

Tue, 10 Jan 2006 00:00:00 +0000

This is part 14/14 of my Implementing IXmlWriter post series.

Today I will add support for writing the generated XML to a C++ stream to last time’s IXmlWriter.

Finally the reason why I’ve insisted on calling this series IXmlWriter (instead of StringXmlWriter) should become clear: I’ve been planning on supporting writing the generated XML to more than just a string. Specifically, today I will add the ability to write the XML to a C++ ostream object, a base class in the C++ iostream library which defines a writable stream.

Implementing IXmlWriter Part 13: Putting IXmlWriter Behind A Pimpl Firewall

Thu, 15 Dec 2005 00:00:00 +0000

This is part 13/14 of my Implementing IXmlWriter post series.

As the private members of IXmlWriter are getting too numerous and too likely to change by my judgment, today I will put last time’s IXmlWriter behind a compilation firewall (pimpl).

The idea behind the pimpl idiom is to hide as much of the class definition as possible in order to avoid requiring users of the class to recompile if the class’s private members are changed. It is accomplished by moving all private members (functions, data, etc.) into a separate class (called the implementation or pimpl class) hidden from the class definition, and replacing these members with an opaque pointer to a forward declaration of this class. It works because a C++ compiler does not need to have the full definition of a class visible in order to allocate space for a pointer to the class; every pointer is a constant, fixed size (often 4 bytes).

Implementing IXmlWriter Part 12: Supporting Pretty-Printing

Tue, 13 Dec 2005 00:00:00 +0000

This is part 12/14 of my Implementing IXmlWriter post series.

Today I will add support for pretty-printing to last time’s IXmlWriter.

Pretty-printing is the addition of whitespace at predetermined locations to make the resulting XML easier to read than when it is all on one line. In the .NET Framework’s System.Xml.XmlTextWriter class, it is supported by the properties Formatting, which allows you to enable or disable pretty-printing; Indentation, which allows you to specify how many whitespace characters indentation should use; and IndentChar, which allows you to specify the whitespace character to use for indentation. For IXmlWriter, I instead chose to expose these features exclusively through the constructor. This frees me from the worry of a user trying to change these properties after IXmlWriter has already begun writing XML, which could produce awkward results. Default parameters are used to make the use of pretty-printing optional and straightforward.

Implementing IXmlWriter Part 11: Supporting Namespaces

Wed, 07 Dec 2005 00:00:00 +0000

This is part 11/14 of my Implementing IXmlWriter post series.

Today I will add support for namespaces to last time’s IXmlWriter.

Namespaces are defined by the W3C recommendation Namespaces in XML. Using namespaces requires two parts: a namespace declaration, which associates a prefix with a namespace name (a user-defined, ideally globally-unique string which defines the namespace, often in the form of a URL); and the assignment of XML elements and attributes to this namespace by using the aforementioned prefix.

Implementing IXmlWriter Part 10: Supporting WriteComment()

Fri, 02 Dec 2005 00:00:00 +0000

This is part 10/14 of my Implementing IXmlWriter post series.

Today I will add support for the function WriteComment() to last time’s IXmlWriter.

Quoting from Section 2.5: Comments of the XML 1.0 spec:

Comments MAY appear anywhere in a document outside other markup; in addition, they MAY appear within the document type declaration at places allowed by the grammar.

Considering this, we should allow writing comments in virtually every WriteState that the IXmlWriter can be in. In fact, some quick thought confirms that we should allow it for every WriteState but WriteState_Attribute, as a comment cannot be legally represented between the quotation marks which delimit an attribute value. With this in mind, here’s the test case I wrote:

Implementing IXmlWriter Part 9: Supporting WriteStartDocument() and WriteEndDocument()

Tue, 22 Nov 2005 00:00:00 +0000

This is part 9/14 of my Implementing IXmlWriter post series.

Today I will add support for the functions WriteStartDocument() and WriteEndDocument() to last time’s IXmlWriter.

WriteStartDocument() writes the XML declaration (i.e. <?xml version="1.0"?>) and WriteEndDocument() closes all open attributes and elements and sets the IXmlWriter back in the initial state. Adding support for these functions is straightforward. Note that I have introduced a new IXmlWriter state called WriteState_Prolog; this will be important later.

Implementing IXmlWriter Part 8: Supporting WriteStartAttribute() and WriteEndAttribute()

Fri, 18 Nov 2005 00:00:00 +0000

This is part 8/14 of my Implementing IXmlWriter post series.

Today I will add support for the functions WriteStartAttribute() and WriteEndAttribute() to last time’s IXmlWriter.

These functions are (obviously) used to denote the start and end of an attribute; the attribute value is written using WriteString() (this usage is analogous to WriteStartElement() and WriteEndElement()). Because WriteString() must now be aware of whether it is writing an attribute value or element content, I must keep track of the state the IXmlWriter is in — a change that affects nearly every function.

Implementing IXmlWriter Part 7: Cleaning Up

Thu, 17 Nov 2005 00:00:00 +0000

This is part 7/14 of my Implementing IXmlWriter post series.

Wow, I can’t believe that it’s been over a month already since my last IXmlWriter post. I guess my vacation ruined my exercise plan and my blogging habits. It’s well past time to get back into both.

Rather than introduce a new test case, I’m going to spend today “cleaning up” the previous version of IXmlWriter.

The first cleanup method is trivial but overdue — I will separate the implementation of IXmlWriter from its interface, as a user of IXmlWriter shouldn’t be particularly concerned about its implementation. In other words, I will separate it into a .h and a .cpp file. For now, IXmlWriter will continue to expose some implementation details (e.g. its private members), but these details should change relatively infrequently. If I ever need to completely separate its implementation from its interface, I will consider making it into a COM-like object or using the Pimpl idiom.

Implementing IXmlWriter Part 6: Escaping Attribute Content

Wed, 12 Oct 2005 00:00:00 +0000

This is part 6/14 of my Implementing IXmlWriter post series.

Last time’s IXmlWriter has a serious bug: it doesn’t properly handle attribute value escaping and can lead to malformed XML.

Consider the following test case:

StringXmlWriter xmlWriter;
xmlWriter.WriteStartElement("root");
  xmlWriter.WriteStartElement("element");
    xmlWriter.WriteAttributeString("att", "\"");
  xmlWriter.WriteEndElement();
xmlWriter.WriteEndElement();

std::string strXML = xmlWriter.GetXmlString();

The previous version of IXmlWriter will generate the XML string <root><element att="""/></root>, which is invalid and will be rejected by a XML parser. The rules for XML attribute escaping are given by Section 2.3 of the XML 1.0 spec—specifically, the AttValue literal:

Implementing IXmlWriter Part 5: Supporting WriteAttributeString()

Tue, 11 Oct 2005 00:00:00 +0000

This is part 5/14 of my Implementing IXmlWriter post series.

Today I will add support for writing attributes to yesterday’s version of IXmlWriter.

To learn more about attributes, see the W3C XML 1.0 Recommendation. Writing attributes will be supported with the function WriteAttributeString().

Here’s today’s test case:

StringXmlWriter xmlWriter;
xmlWriter.WriteStartElement("root");
  xmlWriter.WriteStartElement("element");
    xmlWriter.WriteAttributeString("att", "value");
  xmlWriter.WriteEndElement();
xmlWriter.WriteEndElement();

std::string strXML = xmlWriter.GetXmlString();
// strXML should be <root><element att="value"/></root>

Because the changes in Implementing IXmlWriter Part 4 keep start elements unclosed until another function is called which requires them to be closed (e.g. WriteString() and WriteEndElement()), adding support for writing attributes is very simple. Here’s the version I came up with to pass all test cases:

Implementing IXmlWriter Part 4: Collapsing Empty Elements

Mon, 10 Oct 2005 00:00:00 +0000

This is part 4/14 of my Implementing IXmlWriter post series.

One of the enhancements that XML introduced over SGML was a shorthand for specifying an element with no content by adding a trailing slash at the end of an open element. For example,   is equivalent to  . Let’s add this functionality to the previous version of IXmlWriter.

Here’s the test case:

StringXmlWriter xmlWriter;
xmlWriter.WriteStartElement("root");
  xmlWriter.WriteStartElement("emptyElement");
  xmlWriter.WriteEndElement();
xmlWriter.WriteEndElement();

std::string strXML = xmlWriter.GetXmlString();
// strXML should be <root><emptyElement/></root>

How does this affect our previous implementation?

Implementing IXmlWriter Part 3: Supporting WriteElementString()

Fri, 07 Oct 2005 00:00:00 +0000

This is part 3/14 of my Implementing IXmlWriter post series.

Today’s addition to the previous iteration of IXmlWriter is quite trivial: supporting the WriteElementString() method.

Here’s the test case:

StringXmlWriter xmlWriter;
xmlWriter.WriteStartElement("root");
  xmlWriter.WriteElementString("element", "value");
xmlWriter.WriteEndElement();

std::string strXML = xmlWriter.GetXmlString();
// strXML should be <root><element>value</element></root>

Implementation is extremely simple because WriteElementString() is nothing but a convenience method which calls WriteStartElement(), WriteString(), and WriteEndElement(). Therefore, here’s the new StringXmlWriter:

class StringXmlWriter
{
private:
    std::stack<std::string> m_openedElements;
    std::string m_xmlStr;

public:
    void WriteStartElement(const std::string& localName)
    {
        m_openedElements.push(localName);
        m_xmlStr += '<';
        m_xmlStr += localName;
        m_xmlStr += '>';
    }

    void WriteEndElement()
    {
        std::string lastOpenedElement = m_openedElements.top();
        m_xmlStr += "</";
        m_xmlStr += lastOpenedElement;
        m_xmlStr += '>';
        m_openedElements.pop();
    }

    void WriteString(const std::string& value)
    {
        typedef std::string::const_iterator iter_t;
        for (iter_t iter = value.begin(); iter != value.end(); ++iter) {
            if (*iter == '&') {
                m_xmlStr += "&amp;";
            } else if (*iter == '<') {
                m_xmlStr += "&lt;";
            } else if (*iter == '>') {
                m_xmlStr += "&gt;";
            } else {
                m_xmlStr += *iter;
            }
        }
    }

    void WriteElementString(const std::string& localName,
                            const std::string& value)
    {
        WriteStartElement(localName);
        WriteString(value);
        WriteEndElement();
    }

    std::string GetXmlString() const
    {
        return m_xmlStr;
    }
};

Implementing IXmlWriter Part 2: Escaping Element Content

Thu, 06 Oct 2005 00:00:00 +0000

This is part 2/14 of my Implementing IXmlWriter post series.

In the previous post of this series, we ended up with a simple class which could write XML elements and element content to a std::string. However, this code has a common, serious problem that was mentioned in my post Don’t Form XML Using String Concatenation: it doesn’t properly escape XML special characters such as & and <. This means that if you call WriteString() with one of these characters, your generated XML will be invalid and will not be able to be parsed by an XML parser.

Implementing IXmlWriter Part 1: The Basics

Fri, 30 Sep 2005 00:00:00 +0000

This is part 1/14 of my Implementing IXmlWriter post series.

After writing my blog post Don’t Form XML Using String Concatenation, I realized that writing a C++ System.Xml.XmlWriter workalike involves some interesting challenges. Therefore, I’ve decided to write a series of blog posts about building a streaming C++ XML generator, a.k.a. IXmlWriter, step-by-step. For this series, I will follow the practice of test-driven development and write a test case followed by an implementation which passes the test case. Future posts’ test cases will be constructed to illustrate bugs in or new features desired from the previous post’s implementation. The test cases will be constructed with the goal of having IXmlWriter be as similar to System.Xml.XmlWriter as possible.

Don't Form XML Using String Concatenation

Fri, 16 Sep 2005 00:00:00 +0000

It seems very common for developers to create XML using string concatenation, as in:

std::string CreateXML
    (
    const std::string& strValue
    )
{
    std::string strXML("<tag>");
    strXML += strValue;
    strXML += "</tag>";
    return strXML;
}

As any experienced XML developer knows, this code has a bug: strValue must be escaped (& must be converted to &, < must be converted to <, etc.) or the XML that is generated will not be well-formed and will not be able to be parsed by an XML parser. One could write a simple function to handle this escaping, but there are so many other issues that can potentially creep up with XML generation—string encoding, illegal element and attribute names, collapsing empty elements, making sure all opened elements are closed, the performance of string concatenation, etc.—that I recommend putting all this logic into a single class. In fact, I suggest modelling the class after .NET’s streaming XML generator, System.Xml.XmlWriter.

Avoid Atoi() And Similar Functions

Tue, 13 Sep 2005 00:00:00 +0000

Avoid atoi() and related functions: they do not distinguish between invalid input and a valid “0″ string. Use functions which properly report errors, such as strtod().

Be Careful With Doubles And C++ Streams

Mon, 12 Sep 2005 00:00:00 +0000

I ran across a piece of code recently that was using ostrstream to convert a double to a string. The code looked something like:

std::string DoubleToString(double d) {
    std::ostrstream ostr;
    ostr << d << std::ends;
    std::string str(ostr.str());
    ostr.freeze(false);
    return str;
}

This function was used to convert doubles to strings for insertion into an XML document, which were eventually parsed in an XSLT by the XPath number() function. Most of the time it worked fine, but for really large numbers the number() function failed and return NaN. Why?

Use RAII

Fri, 09 Sep 2005 00:00:00 +0000

This is covered by any halfway-decent C++ book, but I believe it deserves reiteration: Use the RAII idiom. I don’t think I could explain RAII any better than HackCraft does in The RAII Programming Idiom.

Let me demonstrate how to use RAII with a semi-contrived example. Here’s the pre-RAII code:

HMODULE hm = LoadLibrary(_T("user32.dll"));
if (hm != NULL) {
    FARPROC proc = GetProcAddress(hm, "MessageBoxW");
    if (proc != NULL) {
        typedef int (WINAPI *FnMessageBoxW)(HWND, LPCWSTR, LPCWSTR, UINT);FnMessageBoxW fnMessageBoxW = (FnMessageBoxW) proc;
        fnMessageBoxW(NULL, L"Hello World!", L"Hello World", MB_OK);
    }
    FreeLibrary(hm);
}

In this case, the resource wrapped is HMODULE, the resource acquisition function is LoadLibrary(), and the resource release function is FreeLibrary(). Beware of resources which have multiple resource release functions, such as Win32’s HANDLE with FindClose() and CloseHandle(); for these cases you will typically have to write multiple RAII classes. I will call the wrapper class HModule. Here’s how its use will look:

Beware File Open/Save Dialogs: They May Change The Current Directory

Thu, 08 Sep 2005 00:00:00 +0000

On some Windows operating systems (primarily Windows 95, 98, and ME), GetOpenFileName() and GetSaveFileName() (and wrappers of these functions such as MFC’s CFileDialog) will permanently change the process’s current working directory unless the OFN_NOCHANGEDIR option is specified. As you can imagine, this can easily break your application if you ever rely on the current working directory being set to a particular value (such as if you open files using relative paths).

Of course, it is best to eliminate any such current working directory assumptions from your application completely.

Be Careful With Bitfields

Wed, 07 Sep 2005 00:00:00 +0000

Consider the following piece of code (adapted from a real-world bug):

BOOL IsDirectory(LPCTSTR strPathName)
{
    DWORD dwAtts = GetFileAttributes(strPathName);
    if (dwAtts == INVALID_FILE_ATTRIBUTES)
        return FALSE;
    return (dwAtts == FILE_ATTRIBUTE_DIRECTORY);
}

This function will work most of the time, but every now and again it will run across a directory which it will claim isn’t one. Why?

The problem is caused because dwAtts is a bitfield (see the GetFileAttributes() documentation). Simple equality comparison isn’t appropriate, as if the directory has any other attributes (such as compressed, hidden, offline, etc.) the comparison will fail.

Use Constant References For Input-Only Parameters, Pointers Otherwise

Tue, 06 Sep 2005 00:00:00 +0000

This is a style issue, so there is no right or wrong, but I suggest using a const reference for an input-only paramater to a C++ function and a pointer for an input/output or output-only parameter. This makes it slightly more obvious that the parameter might be modified by the function.

Example:

ReturnType Function
    (
    const ParamType& inputParam,
    ParamType* inputOutputParam,
    ParamType* outputParam
    )
{
    // ...
}

void User()
{
    ParamType inputParam;
    ParamType inputOutputParam;
    ParamType outputParam;

    // Note the &s -- this is a bit of a warning something
    // might happen to inputOutputParam or outputParam...
    ReturnType ret = Function
        (
        inputParam,
        &inputOutputParam,
        &outputParam
        );
}

Write Functions Which Take Iterators, Not Collections

Mon, 05 Sep 2005 00:00:00 +0000

If my experience is typical, this is a very common construct:

ReturnType Function
    (
    const std::vector<T>& container
    )
{
    typedef std::vector<T>::const_iterator iterator_t;
    for (iterator_t iter = container.begin();
         iter != container.end();
         ++iter)
    {
        // Work with *iter
    }
}

The problem with this construct is that you have forced a container choice upon the user of your function. Slightly better, and basically your only choice when interoping with C, is this:

ReturnType Function
    (
    T* array,
    int numItems
    )
{
    for (int i = 0; i < numItems; ++i) {
        // Work with array[numItems]
    }

    // Or perhaps:
    // for (T* pT = array; pT != array + numItems; ++pT) {
    //     Work with *pT
    // }
}

With the above construct you can pass in any container which uses contiguous storage, such as an array or a std::vector (yes, std::vectors are guaranteed to store the data contiguously). Passing a std::vector to the above function looks like:

Prefer Iteration To Indexing

Fri, 02 Sep 2005 00:00:00 +0000

I’ve seen the following STL construct countless times:

std::vector<T> container;
for (int i = 0; i < container.size(); ++i) {
    // Work with container[i]
}

Unless otherwise necessary, it is better to use an STL iterator because it enables you to more easily change the underlying container. You can isolate the code changes required to one line by using typedef, as in:

typedef std::vector<T> container_t;
container_t container;

// Or ::const_iterator as necessary
for (container_t::iterator iter = container.begin();
     iter != container.end();
     ++iter)
{
    // Work with *iter
}

Note that I wrote iter != container.end() as opposed to iter < container.end(). The former only requires an input iterator, while the latter requires a random access iterator—a more complicated iterator type supported by fewer STL containers.

Revisiting Excel Interop

Wed, 24 Aug 2005 00:00:00 +0000

I ran into problem today relating to Excel interop. A coworker made a change to a C# application I wrote and was trying to build it. The program relied on a project which had a reference to the Microsoft Excel 9.0 Object Library which ships with Office 2000. However, the coworker had Office 2003 installed which includes the Excel 11.0 Object Library and not the Excel 9.0 Object Library. Because of this, he could not build the application.

Deterministic Finalization and IDisposable Part 5: Useful IDisposable Class 3: AutoReleaseComObject

Tue, 15 Feb 2005 00:00:00 +0000

This is part 5/5 of my Deterministic Finalization and IDisposable post series.

This is the final example in my series on deterministic finalization in garbage-collected languages and the true motive behind the series: AutoReleaseComObject. The idea behind AutoReleaseComObject is simple: it is nothing but a wrapper around a COM object which calls Marshal.ReleaseComObject() upon Dispose() until the COM object’s reference count is 0 and the object is freed. Here’s the implementation:

Deterministic Finalization and IDisposable Part 4: Useful IDisposable Class 2: AutoDeleteFile

Mon, 14 Feb 2005 00:00:00 +0000

This is part 4/5 of my Deterministic Finalization and IDisposable post series.

I guess my definition of tomorrow is much longer than I thought, but here’s another useful IDisposable class which I shall present without comment: AutoDeleteFile.

using System;
using System.Diagnostics;
using System.IO;

/// <summary>
/// A file wrapper which automatically deletes the file unless Disarm()
/// is called.
/// </summary>
public sealed class AutoDeleteFile : IDisposable
{
    private FileInfo m_underlyingFile;
    private bool m_armed = true;
    private bool m_disposed = false;

    public AutoDeleteFile(FileInfo underlyingFile)
    {
        Debug.Assert(underlyingFile != null);
        m_underlyingFile = underlyingFile;
    }

    ~AutoDeleteFile()
    {
        Dispose(false);
    }

    public FileInfo File
    {
        get { return m_underlyingFile; }
    }

    public void Disarm()
    {
        m_armed = false;
    }

#region IDisposable Members
    public void Dispose()
    {
        Dispose(true);
        GC.SuppressFinalize(this);
    }
#endregion

    private void Dispose(bool disposing)
    {
        if (!m_disposed)
        {
            if (m_armed)
            {
                try
                {
                    m_underlyingFile.Delete();
                }
                catch (Exception)
                {
                    // If we can't delete, oh well!
                }
            }

            m_disposed = true;
        }
    }
}

Deterministic Finalization and IDisposable Part 3: Useful IDisposable Class 1: TimedLock

Sun, 13 Feb 2005 00:00:00 +0000

This is part 3/5 of my Deterministic Finalization and IDisposable post series.

For the first example of a useful custom class which implements IDisposable, I will simply link to and reproduce Ian Griffith’s TimedLock — an enhancement of the C# lock statement which allows the specification of a timeout period instead of blocking forever while trying to obtain the lock.

The code for TimedLock is reproduced below:

using System;
using System.Threading;

// Thanks to Eric Gunnerson for recommending this be a struct rather
// than a class -- avoids a heap allocation.
// Thanks to Change Gillespie and Jocelyn Coulmance for pointing out
// the bugs that then crept in when I changed it to use struct...
// Thanks to John Sands for providing the necessary incentive to make
// me invent a way of using a struct in both release and debug builds
// without losing the debug leak tracking.
public struct TimedLock : IDisposable
{
    public static TimedLock Lock (object o)
    {
        return Lock (o, TimeSpan.FromSeconds (10));
    }

    public static TimedLock Lock (object o, TimeSpan timeout)
    {
        TimedLock tl = new TimedLock (o);
        if (!Monitor.TryEnter (o, timeout))
        {
#if DEBUG
            System.GC.SuppressFinalize(tl.leakDetector);
#endif
            throw new LockTimeoutException ();
        }
        return tl;
    }

    private TimedLock (object o)
    {
        target = o;
#if DEBUG
        leakDetector = new Sentinel();
#endif
    }

    private object target;

    public void Dispose ()
    {
        Monitor.Exit (target);
        // It's a bad error if someone forgets to call Dispose,
        // so in Debug builds, we put a finalizer in to detect
        // the error. If Dispose is called, we suppress the
        // finalizer.
#if DEBUG
        GC.SuppressFinalize(leakDetector);
#endif
    }

#if DEBUG
    // (In Debug mode, we make it a class so that we can add a finalizer
    // in order to detect when the object is not freed.)
    private class Sentinel
    {
        ~Sentinel()
        {
            // If this finalizer runs, someone somewhere failed to
            // call Dispose, which means we've failed to leave
            // a monitor!
            System.Diagnostics.Debug.Fail("Undisposed lock");
        }
    }

    private Sentinel leakDetector;
#endif
}

public class LockTimeoutException : ApplicationException
{
    public LockTimeoutException () : base("Timeout waiting for lock")
    {
    }
}

It is trivial to use TimedLock instead of lock in your applications. Simply change statements from:

Deterministic Finalization and IDisposable Part 2: Rules For Implementing IDisposable

Sat, 12 Feb 2005 00:00:00 +0000

This is part 2/5 of my Deterministic Finalization and IDisposable post series.

Unfortunately the content of this blog post has been lost.

Deterministic Finalization and IDisposable Part 1: The Basics

Fri, 11 Feb 2005 00:00:00 +0000

This is part 1/5 of my Deterministic Finalization and IDisposable post series.

This topic has been covered many times by many others (such as here and here), so if you are familiar with C#’s using statement and IDisposable interface, feel free to skip this post. I’m writing this introduction to provide the necessary background information to set up a series of subsequent posts.

Garbage collection, found in languages such as C# and Java (among many others), is a very useful feature: it largely alleviates the need for a programmer to manually handle resource management. The most commonly cited benefit is that garbage collection eliminates the need for the programmer to explicitly call heap memory management functions such as malloc and free; instead, the garbage collector automatically keeps track of whether objects are still in use and frees them when they are no longer needed.¹ However, in addition to handling memory management, garbage collection may also release other scarce resources upon cleanup, such as file locks or network connections.

In-Memory Decompression of gzip using Zlib

Wed, 01 Sep 2004 00:00:00 +0000

At work I recently was given the task to support gzip-based HTTP response compression in a C++ MFC application. For a while I was convinced that there was no way to support in-memory decompression of gzip-compressed data using zlib, and so I wrote the gzipped data to a temporary file and then used the gzopen()/gzread()-family of functions in zlib to read it back (as described in the zlib manual).

I spent a lot of time looking at the zlib code to try to see why the in-memory streaming decompression function inflate() wasn’t working for gzip-compressed data and determined that zlib wasn’t recognizing the gzip headers. The inflate() function had code which purported to understand the headers, but it wasn’t being enabled due to reasons unknown. After some more sleuthing, I noticed the gzopen()/gzread() functions implemented their own gzip header detection and passed only the raw data to inflate(). Once I noticed this, I threw my hands up in disgust and said it couldn’t be done.

Using the Excel Object Model and Performance

Tue, 20 Jul 2004 00:00:00 +0000

Recently I’ve had to write a bit of code which communicates with Microsoft Excel using its object model. Here are a few things I have learned from this experience.

Interaction with the Excel object model seems to use some kind of inter-process communication with an Excel process that is started behind the scenes. If things are not shut down properly, this Excel process will continue to run indefinitely in the background. Be sure to periodically check the Task Manager for any runaway Excel processes — these typically indicate a bug in your code or incomplete shutdown (perhaps because you chose “Stop Debugging” from the debugger).
If you are using the Excel object model using MFC, follow the example in Microsoft KB Article 178781: HOWTO: Automate Excel Using MFC and Worksheet Functions. Be sure to call COleDispatchDriver::ReleaseDispatch() or use the COleDispatchDriver::m_bAutoRelease member on all relevant objects or the Excel process may never stop.
The Excel object model documentation (which Microsoft KB Article 222101: How To Find and Use Office Object Model Documentation helps you find) is quite horrible, at least as of Office 2000. It is also written exclusively with the VB developer in mind.
To give control of the running, hidden Excel process with which you are interacting to the user, use the following code:

Excel::_Application app;
// Create and work with app...
app.SetVisible(true);
app.SetUserControl(true);

Even if Excel is not visible to the user, Application::Quit() may pop up a hidden dialog asking if the user wants to save the changes that were made through the dialog box. Since the dialog is not visible, Excel will never shut down. To prevent this dialog, either set Application.DisplayAlerts to false or set Workbook.Saved to true for all modified workbooks. The former is preferred.
Each call using the Excel object model is very, very slow, probably as a result of the use of IPC. This means that the typical way one would think of interacting with cell values in Excel — iterating cell-by-cell within a set of nested for loops — is often too slow to be practical. Instead, I work in selections of nRows rows by nCols columns and use a two-dimensional SAFEARRAY. For example:

COleSafeArray rawData;
DWORD rawDataDimensions[2];
rawDataDimensions[0] = nRows;
rawDataDimensions[1] = nCols;
rawData.Create(VT_VARIANT, 2, rawDataDimensions);
// Populate the values of rawData...
// Select a range of size nRows x nCols
Excel::Range range = wksheet.GetRange(varUpperLeftCell, varLowerRightCell);
// Set the cells' values in one call to .SetValue()
// instead of setting individual cell values
range.SetValue(rawData);