Building a New Public API

This year, I embarked on an exciting journey to build a new public API system for one of my clients. The goal was to create a system allowing them to sell access to their valuable dataset to external parties. This project wasn't just about opening up new revenue streams; it was about innovation and expanding the value we could offer to the clients sector.

APIs play a crucial role in modern software systems, facilitating the flow of essential data across various components and platforms. By enabling third parties to build products on top of our infrastructure, we were set to significantly increase the platform's value and reach.

But our vision extended beyond external use. We aimed to create an API that would also serve as an internal standard for data access across the platform. This dual-purpose approach promised to streamline operations and set a new benchmark for efficiency.

Building the Infrastructure

Starting such a project can be daunting, but I've learned that the act of beginning is often the most crucial milestone. Once you have something tangible, it becomes much easier to iterate and improve. However, before diving into coding, I knew that designing the right infrastructure from the start would be half the battle. The old adage "measure twice, cut once" came to mind.

My focus was on three key areas:

1. Technologies: Choosing the right tech stack
2. Structure: Ensuring maintainability through proper project organization
3. Security: Implementing robust measures to protect the main platform

Technologies

Given that the main platform runs on .NET 5/6+, I decided to leverage this technology as the foundation for the new API system. This decision immediately descoped a significant amount of effort – a critical consideration for a solo developer.

I also wanted to incorporate Swagger with its OpenAPI specification, tools I'd had positive experiences with in the past. These two key technologies formed the base upon which I built the rest of the stack.

The final tech stack included:

• .NET
• Swagger + OpenAPI specification
• Microsoft SQL Server
• Docker
• Microsoft entity framework

Project Structure

For the project structure, I opted to create a new area within the existing solution rather than a separate .sln file. This approach maintains a cohesive view of the entire codebase, which is particularly beneficial for smaller teams.

I started by defining the routes for the initial version of the API:

/api/v2/campuses
/api/v2/campuses/[id]
/api/v2/courses
/api/v2/courses/[id]
/api/v2/intakes
/api/v2/intakes/[id]
/api/v2/providers
/api/v2/providers/[id]

Based on these routes, I created the following directory structure:

PublicApi/
├── Controllers/
│   ├── CampusesController.cs
│   ├── CoursesController.cs
│   ├── IntakesController.cs
│   ├── ProviderController.cs
│   └── ScholarshipController.cs
├── Models/
│   ├── Campus.cs
│   ├── Course.cs
│   ├── Intake.cs
│   ├── Provider.cs
│   └── Scholarship.cs
└── Services/
    (initially empty, to be populated as needed)

I began by defining the models that shaped the API's response payloads. With these in place, I could then set up basic GET endpoints for both list and individual resource retrieval, initially returning dummy data.

Here's a basic example of one of the domain-driven endpoints:

using System.Collections.Generic;

namespace PublicApi.Models
{
    public class Course
    {
        public string Name { get; set; }
        public string Id { get; set; }
        public string ProviderId { get; set; }
        public string LevelOfStudy { get; set; }
        public List AreasOfStudy { get; set; }
        public List SubjectsOfStudy { get; set; }
        public string Details { get; set; }
        public string EntryRequirements { get; set; }
        public string Duration { get; set; }
        public bool Active { get; set; }
        public long LastUpdated { get; set; }
        public bool InternationalFlag { get; set; }
    }
}

using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using PublicApi.Models;

namespace PublicApi.Controllers
{
    [Area("PublicApi")]
    [ApiController]
    [Route("api/v2/[controller]")]
    [ApiExplorerSettings(GroupName = "PublicApi")]
    public class CoursesController : ControllerBase
    {
        [HttpGet]
        public ActionResult> GetCourses()
        {
            // Dummy data for illustration
            var courses = new List
            {
                new Course 
                { 
                    Name = "Computer Science",
                    Id = "CS101",
                    ProviderId = "UNIV001",
                    LevelOfStudy = "Undergraduate",
                    AreasOfStudy = new List { "Technology", "Mathematics" },
                    SubjectsOfStudy = new List { "Programming", "Algorithms", "Data Structures" },
                    Details = "A comprehensive course covering the fundamentals of computer science.",
                    EntryRequirements = "High school diploma with strong mathematics background",
                    Duration = "4 years",
                    Active = true,
                    LastUpdated = 1630444800,
                    InternationalFlag = true
                },
                new Course 
                { 
                    Name = "Business Administration",
                    Id = "BA201",
                    ProviderId = "UNIV001",
                    LevelOfStudy = "Graduate",
                    AreasOfStudy = new List { "Business", "Management" },
                    SubjectsOfStudy = new List { "Finance", "Marketing", "Operations" },
                    Details = "An MBA program designed for aspiring business leaders.",
                    EntryRequirements = "Bachelor's degree and 2 years of work experience",
                    Duration = "2 years",
                    Active = true,
                    LastUpdated = 1641024000,
                    InternationalFlag = true
                }
            };

            return Ok(courses);
        }

        [HttpGet("{id}")]
        public ActionResult GetCourse(string id)
        {
            // Dummy data for illustration
            var course = new Course
            {
                Name = "Data Science",
                Id = id,
                ProviderId = "UNIV001",
                LevelOfStudy = "Graduate",
                AreasOfStudy = new List { "Technology", "Statistics" },
                SubjectsOfStudy = new List { "Machine Learning", "Big Data", "Statistical Analysis" },
                Details = "An advanced course in data science and analytics.",
                EntryRequirements = "Bachelor's degree in a quantitative field",
                Duration = "2 years",
                Active = true,
                LastUpdated = 1651363200,
                InternationalFlag = true
            };

            return Ok(course);
        }
    }
}

This setup allowed me to perform my first test using a Bruno client, and voila! I received my first response. Now we're getting somewhere

Security

Before shipping anything, implementing security measures was crucial. I focused on two main concerns:

1. Unauthorized access: I implemented a rough-cut API key authentication strategy and added our new API pages to the robots.txt file to prevent search engine indexing. This would make it difficult to stumble across our API accidentally, and if they did, they wouldn't be able to access our system without a valid API key.
2. Protection against DoS: To mitigate the risk of database read operation overload (intentional or unintentional), I implemented API key rate limiting in combination with IP rate limits. I set a sensible limit of 120 requests per minute (2 requests per second) to maintain a reasonable SLA while protecting the system from accidental request floods.

Making It a Product

While I had a basic API in production, transforming it into a valuable product required several additional steps:

Adding Business Logic

We needed to get the routes wired up so they could start returning something valuable to the user. I used Microsoft's Entity Framework ORM to pull records from the SQL database and map them to the API response payloads. This process involved creating data access layers and implementing the necessary business logic in each controller.

Creating Services for Reusable Logic

To promote code reuse and maintain a clean separation of concerns, I abstracted common business logic into services. By the end of the project, I had created several utility services:

Services/
├── AuthorityFormatter.cs
├── DateFormatCalculator.cs
├── DeliveryCalculator.cs
├── DurationCalculator.cs
├── HtmlHelper.cs
├── MacronRemover.cs
├── ProviderTypeFormatter.cs
├── RegionMapper.cs
├── StreetAddressFormatter.cs
├── StringFormatter.cs
└── SubjectTaxonomyMapper.cs

Implementing Pagination

Pagination was crucial for allowing third parties to navigate through records at the API level efficiently. I created a pagination model and service:

public class ApiPaginatedResponse
{
    public List Items { get; set; }
    public Pagination Pagination { get; set; }
}

public class Pagination
{
    public int Skip { get; set; }
    public int Limit { get; set; }
    public int Count { get; set; }
    public string NextPage { get; set; }
    public int TotalCount { get; set; }
}

Implementing Correct HTTP Response Codes

An often overlooked but crucial aspect of API design is the proper use of HTTP response codes. These codes provide immediate feedback to API consumers (and developers) about the status of their requests, making the API more intuitive and easier to work with.

I made sure to implement a range of appropriate status codes in our API responses:

• 200 OK: For successful GET, PUT, or PATCH requests
• 400 Bad Request: When the request is malformed or contains invalid parameters
• 401 Unauthorized: When authentication is required but not provided or is invalid
• 403 Forbidden: When the authenticated user doesn't have permission to access the requested resource
• 404 Not Found: When the requested resource doesn't exist
• 429 Too Many Requests: When the client has sent too many requests in a given amount of time (rate limiting)
• 500 Internal Server Error: For unexpected server errors

Customizing the UI

To enhance the user experience and bring our brand to the API, I customized the Swagger UI by replacing the generic branding with our logo and applying clean, consistent styling.

I also added example response payloads for various endpoints: 

Swagger/
└───Examples
    ├───CampusExamples
    │       CampusListResponseExample.cs
    │       CampusResponseExample.cs
    │
    ├───CourseExamples
    │       CourseListResponseExample.cs
    │       CourseResponseExample.cs
    │
    ├───IntakeExamples
    │       IntakeListResponseExample.cs
    │       IntakeResponseExample.cs
    │
    ├───ProviderExamples
    │       ProviderListResponseExample.cs
    │       ProviderResponseExample.cs
    │
    ├───ScholarshipExamples
    │       ScholarshipListResponseExample.cs
    │       ScholarshipResponseExample.cs
    │
    └───StatusCodes
            400ResponseExample.cs
            401ResponseExample.cs
            404ResponseExample.cs
            429ResponseExample.cs

Provisioning API Keys

Before sharing the new Swagger documentation, I generated and distributed API keys to the third parties who would be using the new API. This process involved creating a secure system for generating, storing, and managing these keys.

You don't need to overthink this. We assigned API keys to a singleton and added code comments to identify the associated users. This simple system allowed for easy management and potential future revocation of access if needed.

Lessons Learned

Building this project was an exciting challenge that taught me several valuable lessons:

1. Start small and iterate: Beginning with a minimal viable product and improving it continuously proved to be an effective strategy.
2. Define first, then build: Having a clear understanding of the desired responses greatly simplified the process of writing business logic.
3. Don't stress the small stuff: Getting the service into users' hands quickly for feedback is crucial, even if it's not 100% polished.
4. Leverage existing resources: Choosing to use languages and frameworks already in play resulted in a more cohesive solution and easier future maintenance.

Future Horizons

The development of this API has not only expanded our offerings but has also opened up new possibilities for innovation in the education sector. As more organizations recognize the power of APIs in driving growth and fostering ecosystems, projects like this will become increasingly vital.

Whether you're considering building an API for your own project or looking to leverage existing APIs in your organization, remember that the journey of a thousand miles begins with a single step. Start small, focus on delivering value, and don't be afraid to iterate and improve as you go.