🐛 [Bug]: UTF8 not enabled by default

[the-hotmann]

Bug Description

I used Fiber to handle and API Endpoint and take JSON objects with strings and simply store them in a DB.

The strings are getting send as UTF8 encoded (which everything should be).
When I read the DB it shows broken chars like:

M�dchen (should be Mädchen)

The fix was to implement this:

app.Use(func(c *fiber.Ctx) error {
	if c.Is("json") {
		c.Set(fiber.HeaderContentType, fiber.MIMEApplicationJSONCharsetUTF8)
	}
	return c.Next()
})

Which I think is a good workaround, but this just works for JSON and also there was no was just setting the encoding to UTF8 without touching the mimetype.

I think Fiber should work with UTF-8 encoding by default, as it can handle mostly everything.

There should be something like this:

app := fiber.New(fiber.Config{
    Encoding: utf-8mb4, // can be `utf-8`, `utf-8mb4`, `iso-8859`, `windows-1251`, `unicode`, `auto`
})

Default should be utf-8. Mostly it is just important, to have a matching encoding on both sides.
auto could automatically detect the encoding on a request per request base - maybe?

How to Reproduce

1. use fiber to store JSON responses with Umlaute to a DB
2. display the content with Fiber aswell
3. notice the broken special chars

Expected Behavior

I was expecting, that Fiber automatically identified which encoding is used/needed, or automatically goes to the most powerful encoding, so nothing breaks.

Code Snippet (optional)

app.Use(func(c *fiber.Ctx) error {
	if c.Is("json") {
		c.Set(fiber.HeaderContentType, fiber.MIMEApplicationJSONCharsetUTF8)
	}
	return c.Next()
})

### Checklist:

- [X] I agree to follow Fiber's [Code of Conduct](https://github.com/gofiber/fiber/blob/master/.github/CODE_OF_CONDUCT.md).
- [X] I have checked for existing issues that describe my problem prior to opening this one.
- [X] I understand that improperly formatted bug reports may be closed without explanation.

[the-hotmann]

Just want to add some more info.

For me it would make sense, if the mimetype of responses that are JSON is application/json.

so this:

app.Use(func(c *fiber.Ctx) error {
	if c.Is("json") {
		c.Set(fiber.HeaderContentType, fiber.MIMEApplicationJSON)
	}
	return c.Next()
})

could (or maybe is) the default.
The Encoding in the config should therefore just add the Encoding, and not overwrite the whole mimetype.
I dont know if that is possible. But maybe something like this.

app.Use(func(c *fiber.Ctx) error {
	c.Set(fiber.Encoding, fiber.utf-8mb4)
	return c.Next()
})

could be the direct implementation, without the config. Anyway I would prefer to set it in the config.
This is just if someone wants to set it explicitely, or with an presettet if-Statement. (like above, with the c.Is("json"))

[gaby]

@MartinHotmann The JSON() function doesnt set the encoding, which probably needs to be updated here: https://github.com/gofiber/fiber/blob/master/ctx.go#L856

Looking at fasthttp, they have SetContentEncoding() which we could expose/call from Fiber to allow the user to the encoding

[gaby]

Question, why is using fiber.MIMEApplicationJSONCharsetUTF8 not a viable option in your use-case when returning back to the user?

[the-hotmann]

Question, why is using fiber.MIMEApplicationJSONCharsetUTF8 not a viable option in your use-case when returning back to the user?

In my limited usecase it actually is an option - thats why I did it.
But once not just UTF-8 encoded strings are send, it is again broken.
Also this is for JSON only and I in general would love to see a global implementation of encoding support.
In general I would like to use encoding UTF8mb4 for the whole app, and for some special endpoints I would like to even switch to unicode.

But like I said, fiber.MIMEApplicationJSONCharsetUTF8 is MimeType + Encoding. It is not possible to handle encoding/charset seperately and properly (in my opinion).

Even if a Default Encoding/Charset is set for JSON for example, the global config should still be able to overwrite it (but just the encoding, not the MimeType).

I just encountered this lately and thought that receiving & sending strings is a comming thing for a API first webframework and therefore probably others are also running into this problem. Before everyone fixed it on its own and in its own way, a global config option could solve it more elegantly.

[ReneWerner87]

https://www.rfc-editor.org/rfc/rfc8259

Question, why is using fiber.MIMEApplicationJSONCharsetUTF8 not a viable option in your use-case when returning back to the user?

In my limited usecase it actually is an option - thats why I did it. But once not just UTF-8 encoded strings are send, it is again broken. Also this is for JSON only and I in general would love to see a global implementation of encoding support. In general I would like to use encoding UTF8mb4 for the whole app, and for some special endpoints I would like to even switch to unicode.

But like I said, fiber.MIMEApplicationJSONCharsetUTF8 is MimeType + Encoding. It is not possible to handle encoding/charset seperately and properly (in my opinion).

Even if a Default Encoding/Charset is set for JSON for example, the global config should still be able to overwrite it (but just the encoding, not the MimeType).

I just encountered this lately and thought that receiving & sending strings is a comming thing for a API first webframework and therefore probably others are also running into this problem. Before everyone fixed it on its own and in its own way, a global config option could solve it more elegantly.

@the-hotmann
which functions and places do you think we should adjust with this global setting? only the ctx.JSON or which other functions?

[the-hotmann]

Ah glad they recommend it aswell.

I don't know all the functions in which it needs to be adjusted, but let me give you another explanation of the problem, maybe that solves the question.

The flow is like this:

TYPE	INPUT (JSON)	STORAGE (DB)	RENDER (JSON)
default	Mädchen	M�dchen	M�dchen
fix	Mädchen	Mädchen	Mädchen

So I assume, the problem occures when receiving the UTF-8 JSON. As Fiber receives it and does not encode it properly.

I guess adjusting the ctx.JSON would fix it. But why not encoding all and everything by default as UTF-8, as this is the standard?
Probably the same will happen when receiving XML or BSON etc - I did not test but would assume this is the case at all other formats aswell.

UTF-8 handles mostly everything correct, even things that are not UTF-8.

[ReneWerner87]

As Fiber receives it and does not encode it properly.

Are you sure about this
What I marked should actually mean that json is in utf8 by default

Can you prepare an example where it is shown that fiber does not encode the json correctly

Curl request + golang api

[the-hotmann]

Thanks, I will try to make one and see if it really is like this. Will post it here.

[the-hotmann]

I could not reproduce it with curl. But the issue occured with a PS1 script.
When sending JSON with a PS1 it broke the encoding. But maybe the problem lays there.

Here a quick demo of the code I use:

package main

import (
	"fmt"

	"github.com/gofiber/fiber/v2"
)

type TEST struct {
	Text string `json:"text"`
}

func main() {
	app := fiber.New()

	app.Post("/eingang", func(c *fiber.Ctx) error {

		var meinTest TEST

		if err := c.BodyParser(&meinTest); err != nil {

			return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
				"fehler": "Ungültiges JSON",
			})

		}

		fmt.Printf("Empfangenes JSON: %+v\n", meinTest)

		return c.JSON(fiber.Map{
			"nachricht": "JSON erfolgreich empfangen",
		})

	})

	app.Listen(":8003")
}

curl:

curl -X POST -H "Content-Type: application/json" -d '{"text":"Überprüfe: »«Ø®"}' http://localhost:8003/eingang

output curl:

Empfangenes JSON: {Text:Überprüfe: »«Ø®}

ps1:

$endpointUrl = "http://localhost:8003/eingang"

$jsonTest = @{
    text = "Überprüfe: »«Ø®"
} | ConvertTo-Json

Write-Host $jsonTest

$headers = @{
    "Content-Type" = "application/json; charset=utf-8"
}

try {
    $antwort = Invoke-RestMethod -Uri $endpointUrl -Method Post -Headers $headers -Body $jsonTest


    Write-Host "Server Antwort:"
    Write-Host $antwort
} catch {
    Write-Host "Fehler: $_"
}

output ps1:

Empfangenes JSON: {Text:�oberprüfe: »«�~®}

(the small ü works, the big one Ü not!? Also some special chars work, some not.)

I printed the string prior sending, it shows this:

{ "text": "Überprüfe: »«Ø®" }

Anyway, adding this:

app.Use(func(c *fiber.Ctx) error {
	if c.Is("json") {
		c.Set(fiber.HeaderContentType, fiber.MIMEApplicationJSONCharsetUTF8)
	}
	return c.Next()
})

Did not change the behaviour. (tested on v2.49.1)

[cardboard-q]

Sorry to jump in halfway, but as an extension to this, in my experience this is based on the client receiving the json, not necessarily fiber itself. Previous discussion: #248

package main

import (
	"github.com/gofiber/fiber/v2"
)

func main() {
	app := fiber.New()
	symbols := "₹£€$₩¥"
	response := fiber.Map{
		"symbols": symbols,
	}

	app.Get("/", func(c *fiber.Ctx) error {
		return c.JSON(response)
	})

	app.Get("/utf8", func(c *fiber.Ctx) error {
		c.JSON(response)
		c.Set("Content-Type", "application/json; charset=utf-8")
		return nil
	})

	app.Listen(":8003")
}

Results on the "/" endpoint:

Expected:
{"symbols":"₹£€$₩¥"}

Chrome/cURL
{"symbols":"₹£€$₩¥"}

Safari:
{"symbols":"₹£€$₩¥"}

Results on the "/utf8" endpoint:

• Receives expected on all endpoints
  • {"symbols":"₹£€$₩¥"}

ctx.JSON will set the content type to application/json without specifying encoding. This can lead to client-side undefined behavior based on if they follow the RFC or not. As a way to mitigate undefined behavior, having the UTF-8 encoding by default would be nice, but I'm not sure it's exactly a bug. A solution requires setting the content-type yourself after calling ctx.JSON as in #248 (and also in the above example) or using a func as in the top of this issue.

[ReneWerner87]

is the problem that the client cannot handle the characters or that the content that goes into the json methodic is not uft8?

[the-hotmann]

Tbh, I dont know anymore. A month ago or so I deleted the whole application.

I will close this for now, but I will do some re-testing once there is an early version of v3 available.