Build a .NET console app for real-world data processing

Console applications are often the least photogenic part of a developer’s toolkit, yet they are excellent workers: they start, process a defined set of files, and finish without asking a browser to carry the load. In Australian teams, a scheduled .NET job can quietly reconcile retailer takings before opening, prepare council data after hours, or turn supplier exports into a finance-ready batch. The interface is small, the boundaries are clear, and the result is easy to automate.

I use this approach for a recurring reconciliation task: several CSV exports arrive, each with its own reference format and quality problems; the application normalises the records, applies business rules, writes accepted rows to SQL Server, and produces a rejection file for follow-up. That example is deliberately ordinary. The techniques transfer to inventory adjustments, membership renewals, payroll exceptions, and migration checks. The goal is a dependable command-line tool with observable inputs, predictable outputs, and an audit trail, rather than a grand rewrite.

Frame a small, measurable data task

Begin with one bounded job whose input, output, and failure conditions can be stated plainly. For example, the application might read daily transaction exports, reject duplicate references, convert values into a common format, and append valid records to a reporting table. Avoid starting with a vague promise to process “all the data”. A precise contract makes every later design decision easier to test.

Define what a successful run means in measurable terms. Record how many source rows were read, how many were accepted, how many were rejected, and which total values were produced. Australian exports commonly contain amounts with GST, Australian dollars, and dates such as 31/03/2025. The specification should state whether a date represents the transaction date or its file date, and whether stored amounts include GST.

Keep the first release focused on one source format and one destination. A useful initial contract might accept a named CSV file, validate its required columns, write accepted and rejected records, and return a non-zero exit code when the input cannot be processed. Add another format only when a real file requires it. This keeps the application understandable while its operational behaviour remains precise.

Set up the solution for change

Create the solution, production project, and test project from the command line. Keeping the projects separate gives the processing code a clean test boundary and makes it possible to publish only the console application.

dotnet new console -n LedgerProcessor
dotnet new xunit -n LedgerProcessor.Tests
dotnet new sln -n LedgerProcessor
dotnet sln LedgerProcessor.sln add LedgerProcessor/LedgerProcessor.csproj LedgerProcessor.Tests/LedgerProcessor.Tests.csproj
dotnet add LedgerProcessor.Tests/LedgerProcessor.Tests.csproj reference LedgerProcessor/LedgerProcessor.csproj

Inside the production project, use folders that reflect responsibilities rather than a large architectural vocabulary. A small application might contain Domain for records and rules, Infrastructure for CSV and SQL Server access, and Services for pipeline orchestration. The generic host supplied by Microsoft.Extensions.Hosting can manage configuration, logging, dependency injection, and graceful cancellation without turning the program into a framework demonstration.

Add packages only when they remove meaningful duplication. CsvHelper can handle quoting and delimiter variations, while Dapper provides a lightweight option for parameterised SQL. Microsoft.Extensions.Logging.Console provides structured logging suitable for scheduled execution. Review current package versions, record them in source control, and restore the same versions during local development and continuous integration.

Parse source files without losing meaning

Treat the source file as untrusted input and convert it into a small domain model. A useful record might contain only the fields required by the business rules, with raw values handled separately during mapping.

public sealed record LedgerEntry(
    DateOnly TransactionDate,
    string Reference,
    decimal Amount);

Use DateOnly for a calendar date and DateTimeOffset when the actual time and offset matter. Parsing should use an explicit contract instead of the workstation’s current culture. An Australian file containing 03/04/2025 needs an agreed meaning, while an amount such as 1,234.56 should be parsed with invariant rules and converted to the correct application type. Preserve source text only when it is needed for diagnostics or audit records.

Validation belongs beside parsing, but it should produce useful evidence. Every rejected row can include the source file, row number, failed rule, and a safe reason. Redact sensitive values rather than copying complete records into logs. The Australian Privacy Act and the Notifiable Data Breaches scheme make careful handling of customer and employee information a real operational requirement, particularly when the application handles tax-file numbers or identity documents. Mike Meredith’s practical testing advice is a useful reference when deciding which parsing behaviours deserve permanent tests.

Build the processing pipeline

Organise the run into explicit stages: load, normalise, validate, apply business rules, and write. Each stage should receive values from the previous stage and return either a result or a structured failure. This keeps the orchestration readable and allows the expensive parts, such as database writes, to sit behind a small interface that tests can replace.

Make the operation idempotent. A scheduled task may be retried after a network failure or started twice by an operator, so writing the same export twice should not create duplicate records. Give each run a batch identifier, calculate a SHA-256 checksum for each source file, and enforce a unique business key such as source system plus transaction reference. Rejected records need a stable key as well, otherwise a retry can conceal the original failure behind another rejection.

Wrap database changes in a transaction and decide what happens to the rejection report when that transaction fails. The application should produce either a complete accepted set and matching report or an unmistakable failure. Stable exit codes make automation predictable: use zero for a completed run, a distinct code for invalid configuration or unusable input, and another for an unexpected failure. I also published the Clubhouse announcement after appearing on a developer show; the useful lesson was to define observable outputs before implementation.

Write results safely and auditably

Choose the destination according to how the result will be consumed. CSV is useful for a portable handoff, hand-off to another system, or manual review. SQL Server is appropriate when downstream applications need indexed, queryable data. Supporting both through a small output interface prevents database concerns from leaking into parsing and validation code.

Use parameterised statements for every value that originates in a file. Enforce uniqueness in the database rather than relying only on an application check. For larger batches, load records into a staging table, validate them there, and apply the final changes inside a transaction. This approach provides clearer errors than a long sequence of individual inserts and makes recovery from a partial import more predictable.

Record an audit entry containing the batch identifier, source filename, checksum, row counts, application version, start time, finish time, and final status. Many Australian banks, insurers, retailers, and government organisations maintain Microsoft-oriented estates, including SQL Server deployments in Sydney, Melbourne, and Canberra. Supporting that installed base can be a deliberate commercial advantage. The audit data should identify the run without storing unnecessary customer details.

Test data, not just happy paths

Unit tests should cover the rules that can lose money or corrupt a report. Test valid dates, impossible dates, duplicate references, missing columns, negative amounts, unexpected status values, and numbers near decimal limits. Include files with a byte-order mark, mixed line endings, quoted commas, blank lines, and trailing spaces. These cases are common in exports from older business systems.

Integration tests should exercise the actual SQL Server schema and transaction boundaries. Testcontainers can start a disposable SQL Server instance in CI, while a local container provides a realistic development environment. A teammate’s remark that a sample “looks pretty right” is not a test oracle; replace it with an agreed expected file, an explicit assertion, or a representative production case.

Time handling deserves particular attention in Australia. Store event timestamps in UTC and convert them only for display. Melbourne and Sydney observe daylight saving, Adelaide uses a half-hour offset, and Perth remains on Western Australian Standard Time. Daylight-saving boundaries can expose code that assumes every Australian offset is a whole hour. Testing those transitions prevents a report from moving records into the wrong trading day.

Put the tool into service

Publish a framework-dependent build where a managed .NET runtime is already available, or a self-contained build when the target machine must run without installation. Single-file publishing simplifies deployment, but the larger artefact and platform-specific output need to be considered.

dotnet publish LedgerProcessor/LedgerProcessor.csproj -c Release -r win-x64 --self-contained false
dotnet publish LedgerProcessor/LedgerProcessor.csproj -c Release -r linux-x64 --self-contained true -p:PublishSingleFile=true

Keep environment-specific values outside the binary. A JSON configuration file can hold the input directory, rejection-file path, and connection-string name, while environment variables or the hosting platform can override those values. Use developer secrets locally and a managed secret store in production. Structured logs should include the batch identifier and outcome without exposing connection strings or sensitive row data.

Create a repeatable release pipeline with automated tests, a version stamp, and a changelog. Whether the team uses Azure DevOps or GitHub Actions, the pipeline should publish an immutable artefact, record its checksum, and retain the package according to the organisation’s support policy. A deployment can then be traced back to a source commit and verified against a known checksum.

Pick one recurring task, create the solution today, and run it against a small sanitised data set in CI. Subscribe to Kilt and Code for more practical .NET and developer-journey writing, and share the finished tool with the team that owns the process.