Custom error handling with AuthJS (NextAuth) v5 Beta and a Custom Backend in Next.js

This is a practical guide (based on personal experience) to integrating your backend API with AuthJS (NextAuth) v5 Beta, covering both credential login and Google OAuth, with tips on token handling and backend verification.

We’ll cover:
✅ Setting up email/password login using the Credentials Provider
✅ Using the Google OAuth Provider alongside a backend validation flow
✅ Handling tokens, user sessions, and backend errors cleanly
✅ Protecting routes and wiring it all together

Whether you’re building an internal tool or a production app, this pattern gives you full control over authentication without sacrificing the built-in power of AuthJS.

Credentials Provider (Email & Password)

For this setup I am assuming your backend has a /auth/sign-in endpoint that:
- Accepts email and password via POST
- Returns a user object and an access token on success
- Returns custom errors for client issues (e.g. wrong password) or server errors

Here’s a simplified server action that calls the backend:

// src/lib/actions/auth.actions.ts
"use server"

export const signIn = async (payload: { email: string; password: string }) => {
  try {
    const response = await baseUrl.post("/auth/sign-in", payload);
    const { token, data: userData } = response?.data?.data;

    if (token && userData) {
      const formattedData = {
        token,
        data: userData as IUser, // type assertion to keep NextAuth happy
      };
      return { data: formattedData };
    }
  } catch (error) {
    return { error: getError(error) };
  }
};

🛠 getError is a small utility you can use to cleanly extract messages from Axios, fetch, or native JS errors.

🛠 baseUrl is my axios instance.

Google OAuth Provider (With Backend Verification)

After a successful sign-in with Google via AuthJS, we take the Google token and verify it with our backend.
Here’s how the flow works:
- AuthJS handles the initial Google sign-in.
- We extract the Google ID token from the user profile.
- We send that token to our backend to verify it.
- Backend validates it via Google’s API, then:
✅ If the email exists: returns the user and session token.
🆕 If not: creates the user and returns the session token.

Here’s my server action for that backend call:

// src/lib/actions/auth.actions.ts
"use server"

export const googleLogin = async (payload: any) => {
  try {
    const response = await baseUrl.post("/auth/google/login", payload);
    const { token, data: userData } = response?.data?.data;

    if (token && userData) {
      const formattedData = {
        token,
        data: userData as IUser,
      };
      return { data: formattedData };
    }
  } catch (error) {
    throw new Error(getError(error));
  }
};

With this setup in place, we’re ready to move into the AuthJS configuration and show how to wire everything up in your Next.js project: providers, callbacks, middleware, and client-side usage.

Setting Up AuthJS with Credentials and Google Providers

Here’s how we configure AuthJS (NextAuth v5) to work with both a custom email/password backend and Google OAuth, while still keeping full control of error handling, token flow, and user session.

We’ll be working with a shared auth.ts file and exporting everything from there — as suggested by NextAuth.

// src/auth.ts
import NextAuth, { AuthError } from "next-auth";
import Google from "next-auth/providers/google";
import CredentialsProvider from "next-auth/providers/credentials";

import { googleLogin, signIn as Login } from "@/lib/actions/auth.actions";
import { getError } from "./lib";

/*
NextAuth suppresses error messages by default. 
To get around this, we create a custom AuthError class that lets us pass clean, 
actionable error messages back to the frontend:
*/
export class InvalidLoginError extends AuthError {
  code = "custom";
  errorMessage: string;
  constructor(message?: any, errorOptions?: any) {
    super(message, errorOptions);
    this.errorMessage = message;
  }
}

export const { handlers, signIn, signOut, auth } = NextAuth({
  pages: {
    signIn: "/signin",
    error: "/signin",
  },
  session: {
    strategy: "jwt",
    maxAge: 24 * 60 * 60, // 1 day
  },
  jwt: {
    maxAge: 30 * 24 * 60 * 60, // 30 days
  },
  providers: [
    // Google Provider
    Google({
      clientId: process.env.AUTH_GOOGLE_ID,
      clientSecret: process.env.AUTH_GOOGLE_SECRET,
      authorization: {
        params: {
          prompt: "consent",
          access_type: "offline",
          response_type: "code",
        },
      },
    }),

    // Credentials Provider
    CredentialsProvider({
      name: "Credentials",
      credentials: {
        email: { label: "Email", type: "text" },
        password: { label: "Password", type: "password" },
      },

      async authorize(credentials) {
        const email = credentials?.email as string;
        const password = credentials?.password as string;

        try {
          if (!password || !email) {
            throw new InvalidLoginError("Invalid credentials");
          }

          const response = await Login({ email, password });

          if (response?.error) {
            throw new Error(response?.error);
          }

          return response?.data;
        } catch (error: any) {
          throw new InvalidLoginError(getError(error));
        }
      },
    }),
  ],
  callbacks: {
    // Restrict access to routes based on user session
    /*
      In v4, authorized didn’t exist — route protection was handled via middleware 
      or session checks in getServerSideProps.
      v5 introduces the authorized callback directly inside your 
      Auth configuration for cleaner route-level control.
     */
   authorized: async ({ auth }) => {
      // `auth` is the session info available during route protection
      // Returning `true` means the request is authorized, `false` blocks it.
      // Here we simply check if the user is logged in.
      return !!auth; // Only logged-in users are authorized
    },

    // Handle logic during user sign-in (runs whenever a user logs in)
    async signIn({ user, account }) {
      // If user is signing in with Google OAuth
      if (account?.provider === "google") {
          try {
            // Exchange Google's ID token with your backend for your custom token
            const res = await googleLogin({
              clientId: process.env.AUTH_GOOGLE_ID!,
              select_by: "random string", // optional; identifies the login action
              credential: account.id_token!, // Google-issued ID token
            });
    
            // If backend didn't return a valid token, reject the sign-in
            if (!res?.token) {
              throw new InvalidLoginError("Failed to login with Google");
            }
    
            // Attach backend token and user data to the `user` object
            (user as any).token = res.token;
            (user as any).data = res.data;
    
            return true; // allow sign-in to continue
          } catch (error) {
            // Handle any error that occurs during the Google sign-in process
            throw new InvalidLoginError(getError(error));
          }
      }

      /* If it's a credentials-based login (email/password), 
        just allow sign-in */
      return true;
    },


    // Called whenever a JWT is created or updated
    async jwt({ token, user }) {
      // When user logs in, persist custom token & user data inside JWT
      if ((user as any)?.token) {
        token.token = (user as any)?.token; // backend-issued JWT or session token
        token.data = (user as any)?.data;   // full user profile or partial data
      }
      return token; // Always return the updated token
    },

    // 4️⃣ Control what data is exposed to the frontend session
    async session({ session, token }) {
      // Pass backend token and user info to the session object
      session.token = token.token as string; // backend token (used for API calls)
      session.user = token.data as any;      // user info from your backend
      return session;
    },
  }
});

API Route + Middleware Setup

Instead of duplicating logic in your API route and middleware, you can just reference the shared auth export from above.

// src/app/api/auth/[...nextauth]/route.ts
import { handlers } from "@/auth";

export const { GET, POST } = handlers;

// src/middleware.ts
// AuthJS v5 (Beta) Middleware

export { auth as middleware } from "@/auth";

export const config = {
  matcher: ["/", "/(protected-page-1|protected-page-2)/:path*"],
};


/* 
// ⚠️ This is how you'd probably do this in v4 Middleware
// In v4, you manually wrapped middleware using `withAuth`
// and explicitly checked the token/session.

import { withAuth } from "next-auth/middleware";

export default withAuth({
  // You could restrict routes based on role or session existence
  callbacks: {
    authorized: ({ token }) => !!token, // Only allow if session token exists
  },
});

export const config = {
  // Routes to protect
  matcher: ["/dashboard", "/settings/:path*"],
};
*/

/*
- In v4, `withAuth` came from "next-auth/middleware" and manually wrapped your logic.
- You had to use the JWT token to determine if a user was logged in.
- The middleware was separate from your main NextAuth config, which meant duplication.
- v5 simplifies this by exporting `auth` directly from your main config, 
  ensuring both middleware and API routes share the same logic.
- Make sure to check the Migration guide in the links below
*/

Building a Custom Login Form (Email & Password)

For the credentials login, we create a custom form that:
- Uses controlled inputs (useState)
- Handles async login through a server action wrapper (authenticate)
- Displays toasts for feedback
- Redirects the user on success using router.replace

Here’s a breakdown of the full login component:

"use client";

import toast from "react-hot-toast";
import { SignInOptions } from "next-auth/react";
import { useSearchParams, useRouter } from "next/navigation";
import { useState } from "react";
import Link from "next/link";
import { Input, SubmitButton } from "../atoms"; // Reusable input/button components
import { getError } from "@/lib";
import { authenticate } from "@/lib/actions/auth.actions";

export const LoginForm = () => {
  const router = useRouter();
  const [email, setEmail] = useState("");
  const [password, setPassword] = useState("");

  const searchParams = useSearchParams();
  const callbackUrl = searchParams.get("callbackUrl") || "/";

  const formAction = async () => {
    toast.dismiss();
    try {
      if (!email || !password) {
        throw new Error("Fill all required fields!");
      }

      const credentials: SignInOptions<false> = {
        redirect: false,
        email,
        password,
        redirectTo: callbackUrl,
      };

      const res = await authenticate("credentials", credentials);

      if (res?.error) {
        throw new Error(res.error);
      }

      toast.success("Login successful, redirecting...");
      return router.replace(`${res?.url || "/"}`);
    } catch (error: any) {
      toast.error(getError(error) || "Login failed! Please try again.");
    }
  };

  return (
    <form
      action={formAction}
      className="flex flex-col w-full gap-4 pt-6 mx-auto text-sm text-black lg:mt-auto font-body"
    >
      <div className="flex flex-col w-full gap-4 max-w-[500px]">
        <Input
          name="email"
          type="email"
          id="email"
          label="Email Address"
          value={email}
          onChange={(e) => setEmail(e.target.value)}
          placeholder="user@example.com"
        />

        <Input
          name="password"
          id="password"
          type="password"
          label="Password"
          value={password}
          onChange={(e) => setPassword(e.target.value)}
          placeholder="**********"
        />

        <Link
          href="/forgotpassword"
          className="ml-auto font-medium text-dark w-fit hover:underline"
        >
          Forgot password?
        </Link>
      </div>

      <SubmitButton
        className="rounded-[20px] min-w-full md:min-w-fit max-w-[201px] w-full text-center justify-center my-4"
      >
        Sign In
      </SubmitButton>

      <div className="flex gap-1">
        <p className="text-[#64748B]">Don&apos;t have an account? </p>
        <Link
          href="/signup"
          className="font-semibold text-primary hover:underline"
        >
          Sign up
        </Link>
      </div>
    </form>
  );
};

🧪 Auth Helper: Handling Errors with authenticate()

NextAuth by default swallows detailed errors from the backend when using the Credentials Provider. This makes it hard to show helpful messages like:

❌ “Incorrect email or password”

Instead, we wrap the signIn() call in a small authenticate() helper function that catches custom InvalidLoginError instances (like we defined earlier) and extracts a clean message using our getError() utility.

// src/lib/actions/auth.actions.ts
"use server"
import { SignInOptions, signIn as NextAuthSignIn } from "next-auth/react";
import { getError } from "@/lib";

/* 
This small wrapper:
Lets you surface errors thrown by your authorize() logic 
(e.g. InvalidLoginError)
*/

export async function authenticate(
  provider: string,
  signinOptions: SignInOptions<false> = {}
) {
  try {
   return await NextAuthSignIn(provider, signinOptions);
  } catch (error: any) {
    return {
      error: error?.errorMessage || getError(error),
    };
  }
}

Auth Helper: Handling Errors with authenticate()

As mentioned earlier, we wrap signIn() with a custom authenticate() helper to capture backend errors and cleanly show them on the frontend.

But here’s a subtle detail that trips up a lot of people 👇

Which signIn are we using?

When logging in with credentials, we don’t use the default signIn() from "next-auth/react".

Instead:

• ✅ We use the signIn exported from our auth.ts setup file (i.e. NextAuth().signIn)
• ✅ This supports catching and returning custom errors like InvalidLoginError

For other providers like Google, we fall back to the built-in signIn() from next-auth/reactalthough personally I have never experienced any errors with the OAUTH providers.

Now that sign-in is set up, let’s see how to access session data across your app.

Accessing the Session (Server & Client)

Once users are authenticated, you can access their session from anywhere in your Next.js App Router project.

• On the Server (e.g. pages, layouts, or server actions)

// app/dashboard/page.tsx or in any server component
import { auth } from "@/auth";

const session = await auth();

console.log(session?.user?.email);

This is useful for:

• Protecting routes
• Rendering dynamic UI (e.g. based on roles or permissions)
• Fetching user-specific data
• In Client Components

"use client";
import { useSession } from "next-auth/react";

export const Dashboard = () => {
  const { data: session } = useSession();

  return <p>Hello, {session?.user?.name}</p>;
};

This is perfect for:

• Header components (showing user info)
• Buttons like “Logout”
• Personalizing user-facing UI

Logging Out

To log out, you use the built-in signOut() function:

"use client"

import { signOut } from "next-auth/react";

<button onClick={() => signOut({ callbackUrl: "/signin" })}>
  Logout
</button>

So yea, that’s how I would set up and use NextAuth in a Next.js project with custom backend support.

We covered:

• Credentials sign-in with email and password
• Google login and how I pass the token to the backend for verification
• How I return proper errors using InvalidLoginError
• A small helper (authenticate) to cleanly handle those errors in the UI
• How to access sessions on both server and client
• Protecting routes with middleware
• Logging out with a simple signOut() call

Hope this helps someone out there trying to spin up a similar flow.

Got questions? Feedback? I’d love to hear it.

Also, I’m curious — how do you handle authentication in your React or Next.js projects?

Drop a comment, and let’s share some ideas 👇

🔗 Helpful Links

Here are some useful links if you want to dive deeper or need docs for anything I mentioned:

• Git issue thread that helped me figure it out. Thanks Aaliboyev
• AuthJS (NextAuth) Docs — official docs for everything NextAuth-related
• NextAuth v5 Migration Guide — if you’re coming from v4 like myself.
• Next.js App Router Docs — everything about layouts, server components, and routing
• NextAuth authorize callback (Credentials Provider) – how to use authorize to talk to your backend
• Server Actions in Next.js — official guide on using server actions
• Session handling with auth() – accessing session data on the server
• Middleware in Next.js — protect routes with middleware.ts
• React Hot Toast — clean, flexible toast notifications for feedback

Custom error handling with AuthJS (NextAuth) v5 Beta and a Custom Backend in Next.js was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.