[feature request] loki client: java Library to send logs to Loki

[liguozhong]

Our log here has already caught a central Jstorm through other Agents. Here we can use the java language to do the extension.

We hope to write this data to the loki server, and then provide it to the R&D staff in the loki explore of grafana for fault analysis. .

But searched on github, there is no client searching for java.

go : https://github.com/livepeer/loki-client
java : ????

[liguozhong]

this is a java client demo. it works.

package com.taobao.sla.loki;

import com.alibaba.fastjson.JSON;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.util.EntityUtils;

import java.io.InputStream;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.List;
import java.util.Map;
import java.util.TimeZone;

public class loki {

    public static void main(String[] args) throws Exception {
        Map<String, String> labels = Maps.newHashMap();

        labels.put("app", "buy2");
        labels.put("level", "error");
        labels.put("ip", "192.160.287.111");
        labels.put("env", "dev");
        labels.put("job", "test");

        for (int i = 0; i < 2000; i++) {
            testData(System.currentTimeMillis() - 60 * 1000, labels, "user log in fail " + i);
        }

        labels.put("app", "buy2");
        labels.put("level", "info");
        labels.put("ip", "192.160.287.222");
        labels.put("env", "dev");
        labels.put("job", "test");

        for (int i = 0; i < 2000; i++) {
            testData(System.currentTimeMillis(), labels, "user buy success " + i);
        }

        labels.put("app", "carts2");
        labels.put("level", "warn");
        labels.put("ip", "192.160.287.333");
        labels.put("env", "dev");
        labels.put("job", "test");

        for (int i = 0; i < 2000; i++) {
            testData(System.currentTimeMillis() + 60 * 1000, labels, "user carts empty " + i);
        }

    }

    public static void testData(Long timestemp, Map<String, String> labels, String line) throws Exception {

        if (labels == null || labels.isEmpty())
            throw new IllegalArgumentException("loki client check arg fail ,labels ==null");
        SimpleDateFormat format = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");

        format.setTimeZone(TimeZone.getTimeZone("UTC"));
        Entry e = new Entry();
        String time = format.format(new Date(timestemp));
        e.setTs(time);
        e.setLine(line);


        Stream stream = new Stream();

        String lbs = "{";
        int index = 0;
        for (String labelKey : labels.keySet()) {
            String lavelValue = labels.get(labelKey);
            if (index > 0) {
                lbs += ",";
            }
            lbs += labelKey + "=\"" + lavelValue + "\"";
            index++;
        }
        lbs += "}";

        stream.getEntries().add(e);
        stream.setLabels(lbs);

        PushRequest pushRq = new PushRequest();

        pushRq.getStreams().add(stream);
        sendLoki(pushRq, "2", "1");
    }

    public static void sendLoki(PushRequest pushRequest, String orgid, String uid) throws Exception {

        DefaultHttpClient httpClient = new DefaultHttpClient();
        String url = "http://localhost:3100/loki/api/v1/push";
        HttpPost httpPost = new HttpPost(url);
        httpPost.addHeader("Content-Type", "application/json");
        httpPost.addHeader("X-Scope-OrgID", orgid);
        httpPost.addHeader("X-Scope-UserID", uid);

        StringEntity entity = new StringEntity(JSON.toJSONString(pushRequest), "utf-8");
        entity.setContentEncoding("UTF-8");
        entity.setContentType("application/json");
        httpPost.setEntity(entity);

        HttpResponse response = httpClient.execute(httpPost);
        System.out.println("response code :" + response.getStatusLine());
        if (response.getEntity() == null) {
            System.out.println("send loki success response.getEntity():" + response.getEntity());
            return;
        }
        long length = response.getEntity().getContentLength();
        if (length > 0) {
            String httpResult = EntityUtils.toString(response.getEntity(), "utf-8");
            System.out.println("result :" + httpResult);
        } else {
            System.out.println("send loki success length:" + length);
        }
    }

    public static class PushRequest {
        public List<Stream> Streams = Lists.newArrayList();

        public List<Stream> getStreams() {
            return Streams;
        }

        public void setStreams(List<Stream> streams) {
            Streams = streams;
        }
    }

    public static class Stream {
        String Labels = "";
        List<Entry> Entries = Lists.newArrayList();

        public void setLabels(String labels) {
            Labels = labels;
        }

        public String getLabels() {
            return Labels;
        }

        public List<Entry> getEntries() {
            return Entries;
        }

        public void setEntries(List<Entry> entries) {
            Entries = entries;
        }
    }

    public static class Entry {
        String ts;
        String Line;

        public String getTs() {
            return ts;
        }

        public String getLine() {
            return Line;
        }

        public void setTs(String ts) {
            this.ts = ts;
        }

        public void setLine(String line) {
            Line = line;
        }
    }
}

[cyriltovena]

You should not have an IP in labels :)

It looks good, I think @slim-bean might get started on something similar. Not sure about the ETA but we also need a java client.

[sh0rez]

For the time being, you might get along with one of these workarounds:

• let your application log to files, use promtail to ingest these to loki
• Use a combination of our https://github.com/grafana/loki/tree/master/fluentd/fluent-plugin-grafana-loki and their Java plugin https://docs.fluentd.org/language-bindings/java

Would any of these work for you?

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[slim-bean]

This should stay open

…

On Fri, Nov 15, 2019, 3:26 PM stale[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1158?email_source=notifications&email_token=ACO2RKZQ4POA4WNSAMRSEE3QT4AZBA5CNFSM4JBFKGMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEGTXSY#issuecomment-554515403>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACO2RK3QA4K6INXE4E4VIELQT4AZBANCNFSM4JBFKGMA> .

[zhisheng17]

+1 need a Loki java client, we use Flink consume Kafka data, send to Loki.

[tkowalcz]

Hi there,

I am working on a java client & log4j2 appender for Loki. It's in early days but works on few servers already: https://github.com/tkowalcz/tjahzi.

[liguozhong]

Hi there,

I am working on a java client & log4j2 appender for Loki. It's in early days but works on few servers already: https://github.com/tkowalcz/tjahzi.

thank you！！