Web-Design
Thursday January 14, 2021 By David Quintanilla
Integrating A Dialogflow Agent Into A React Application — Smashing Magazine


About The Writer

Nwani Victory works as a Frontend Engineer at Liferithms.inc from Lagos, Nigeria. After workplace hours, he doubles as a Cloud Engineer looking for methods to make Cloud …
More about
Nwani

In relation to constructing a conversational chat assistant that may very well be used at a small or enterprise degree, Dialogflow would almost certainly be one of many first choices that may present up in your search record — and why wouldn’t it? It gives a number of options akin to the power to course of audio and textual content inputs, present dynamic responses utilizing customized webhooks, hook up with tens of millions of Google-enabled units by utilizing the Google assistant, and a lot extra. However other than its console that’s supplied to design and handle an Agent, how can we create a chat assistant that can be utilized inside our constructed internet purposes, too?

Dialogflow is a platform that simplifies the method of making and designing a pure language processing conversational chat assistant which might course of voice or textual content enter when getting used both from the Dialogflow console or from an built-in internet software.

Though the built-in Dialogflow Agent is briefly defined on this article, it’s anticipated that you’ve an understanding of Node.js and Dialogflow. If you’re studying about Dialogflow for the primary time, this article provides a transparent rationalization of what Dialogflow is and its ideas.

This text is a information on how a constructed a Dialogflow agent with voice and chat help that may be built-in into an internet software with the assistance of an Express.js back-end software as a hyperlink between a React.js Internet software and the Agent on Dialogflow itself. By the tip of the article, it’s best to in a position to join your individual Dialogflow agent to your most well-liked internet software.

To make this information straightforward to observe via, you may skip to whichever a part of the tutorial pursuits you most or observe them within the following order as they seem:

1. Setting Up A Dialogflow Agent

As defined in this article, a chat assistant on Dialogflow known as an Agent and it contains of smaller elements akin to intents, fulfillment, knowledge base and rather more. Dialogflow offers a console for customers to create, practice, and design the dialog circulation of an Agent. In our use case, we’ll restore an agent that was exported right into a ZIP folder after being educated, utilizing the agent Export and Import function.

Earlier than we carry out the import, we have to create a brand new agent that will probably be merged with the agent about to be restored. To create a brand new Agent from the console, a singular identify is required and likewise, a mission on the Google Cloud to hyperlink the agent with. If there isn’t a present mission on the Google Cloud to hyperlink with, a brand new one will be created here.

An agent has been beforehand created and educated to suggest wine merchandise to a consumer based mostly on their funds. This agent has been exported right into a ZIP; you may download the folder here and restore it into our newly created agent from the Export and Import tab discovered within the agent Settings web page.

Restoring a previously exported agent from a ZIP folder
Restoring a beforehand exported agent from a ZIP folder. (Large preview)

The imported agent has been beforehand educated to suggest a wine product to the consumer based mostly on the consumer’s funds for buying a bottle of wine.

Going via the imported agent, we’ll see it has three created intents from the intents web page. One being a fallback intent, used when the Agent doesn’t acknowledge enter from a consumer, the opposite is a Welcome intent used when a dialog with the Agent is began, and the final intent is used to suggest a wine to the consumer based mostly on the quantity parameter inside the sentence. Of concern to us is the get-wine-recommendation intent

This intent has a single enter context of wine-recommendation coming from the Default Welcome intent to hyperlink the dialog to this intent.

“A Context is a system inside an Agent used to regulate the circulation of a dialog from one intent to the opposite.”

Beneath the contexts are the Training phrases, that are sentences used to coach an agent on what sort of statements to anticipate from a consumer. By way of a big number of coaching phrases inside an intent, an agent is ready to acknowledge a consumer’s sentence and the intent it falls into.

The coaching phrases inside our brokers get-wine-recommendation intent (as proven beneath) signifies the wine selection and the value class:

List of available training phrases with get-wine-recommendation intent.
Get-wine-recommendation intent web page displaying the accessible coaching phrases. (Large preview)

Trying on the picture above, we will see the accessible coaching phrases listed out, and the forex determine is highlighted in yellow coloration for every of them. This highlighting is named an annotation on Dialogflow and it’s routinely finished to extract the acknowledged information varieties generally known as an entity from a consumer’s sentence.

After this intent has been matched in a dialog with the agent, an HTTP request will probably be made to an exterior service to get the advisable wine based mostly on the value extracted as a parameter from a consumer’s sentence, via using the enabled webhook discovered inside the Success part on the backside of this intent web page.

We are able to take a look at the agent utilizing the Dialogflow emulator situated in the fitting part of the Dialogflow console. To check, we begin the dialog with a “Hello” message and observe up with the specified quantity of wine. The webhook will instantly be referred to as and a wealthy response much like the one beneath will probably be proven by the agent.

Testing the imported agent agent webhook.
Testing the imported agent’s achievement webhook utilizing the Agent emulator within the console. (Large preview)

From the picture above we will see the webhook URL generated utilizing Ngrok and the agent’s response on the right-hand facet displaying a wine inside the $20 value vary typed in by the consumer.

At this level, the Dialogflow agent has been totally setup. We are able to now get began with integrating this agent into an internet software to allow different customers to entry and work together with the agent with out entry to our Dialogflow console.

Integrating A Dialogflow Agent

Whereas there are different technique of connecting to a Dialogflow Agent akin to making HTTP requests to its REST endpoints, the advisable means to connect with Dialogflow is thru using its official consumer library accessible in a number of programming languages. For JavaScript, the @google-cloud/dialogflow package deal is on the market for set up from NPM.

Internally the @google-cloud/dialogflow package deal makes use of gRPC for its community connections and this makes the package deal unsupported inside a browser setting besides when patched utilizing webpack, the advisable means to make use of this package deal is from a Node setting. We are able to do that by establishing an Categorical.js back-end software to make use of this package deal then serve information to the online software via its API endpoints and that is what we’ll do subsequent.

Setting Up A Node Categorical Software

To arrange an categorical software we create a brand new mission listing then seize the wanted dependencies utilizing yarn from an opened command line terminal.

# create a brand new listing and ( && ) transfer into listing
mkdir dialogflow-server && cd dialogflow-server

# create a brand new Node mission
yarn init -y

# Set up wanted packages
yarn add categorical cors dotenv uuid

With the wanted dependencies put in, we will proceed to arrange a really lean Categorical.js server that handles connections on a specified port with CORS help enabled for the online app.

// index.js
const categorical =  require("categorical")
const dotenv =  require("dotenv")
const cors =  require("cors")

dotenv.config();

const app = categorical();
const PORT = course of.env.PORT || 5000;

app.use(cors());

app.hear(PORT, () => console.log(`🔥  server working on port ${PORT}`));

When executed, the code within the snippet above begins an HTTP server that listens for connections on a specified PORT Categorical.js. It additionally has Cross-origin resource sharing (CORS) enabled on all requests utilizing the cors package deal as an Express middleware. For now, this server solely listens for connections, it can’t reply to a request as a result of it has no created route, so let’s create this.

We now want so as to add two new routes: one for sending textual content information whereas the opposite for sending a recorded voice enter. They may each settle for a POST request and ship the info contained within the request physique to the Dialogflow agent in a while.

const categorical = require("categorical") 

const app = categorical()

app.submit("/text-input", (req, res) => {
  res.standing(200).ship({ information : "TEXT ENDPOINT CONNECTION SUCCESSFUL" })
});

app.submit("/voice-input", (req, res) => {
  res.standing(200).ship({ information : "VOICE ENDPOINT CONNECTION SUCCESSFUL" })
});

module.exports = app

Above we created a separate router occasion for the 2 created POST routes which for now, solely reply with a 200 standing code and a hardcoded dummy response. After we are finished authenticating with Dialogflow, we will come again to implement an precise connection to Dialogflow inside these endpoints.

For the final step in our backend software setup, we mount the beforehand created router occasion created into the Categorical software utilizing app.use and a base path for the route.

// agentRoutes.js

const categorical =  require("categorical")
const dotenv =  require("dotenv")
const cors =  require("cors")

const Routes =  require("./routes")

dotenv.config();
const app = categorical();
const PORT = course of.env.PORT || 5000;

app.use(cors());

app.use("/api/agent", Routes);

app.hear(PORT, () => console.log(`🔥  server working on port ${PORT}`));

Above, we’ve added a base path to the 2 routes two we will take a look at any of them by way of a POST request utilizing cURL from a command line because it’s finished beneath with an empty request physique;

curl -X http://localhost:5000/api/agent/text-response

After profitable completion of the request above, we will anticipate to see a response containing object information being printed out to the console.

Now we’re left with making an precise reference to Dialogflow which incorporates dealing with authentication, sending, and receiving information from the Agent on Dialogflow utilizing the @google-cloud/dialogflow package deal.

Authenticating With Dialogflow

Each Dialogflow agent created is linked to a mission on the Google Cloud. To attach externally to the Dialogflow agent, we authenticate with the mission on the Google cloud and use Dialogflow as one of many mission’s sources. Out of the six available methods to connect with a mission on the google-cloud, utilizing the Service accounts choice is probably the most handy when connecting to a specific service on the google cloud via its consumer library.

Observe: For production-ready purposes, using short-lived API keys are advisable over Service Account keys with a view to cut back the danger of a service account key moving into the flawed palms.

What Are Service Accounts?

Service accounts are a particular sort of account on the Google Cloud, created for non-human interplay, largely via exterior APIs. In our software, the service account will probably be accessed via a generated key by the Dialogflow client library to authenticate with the Google Cloud.

The cloud documentation on creating and managing service accounts offers a wonderful guide to create a service account. When creating the service account, the Dialogflow API Admin position needs to be assigned to the created service account as proven within the final step. This position provides the service account administrative management over the linked Dialogflow agent.

To make use of the service account, we have to create a Service Account Key. The next steps beneath define the right way to create one in JSON format:

  1. Click on on the newly created Service Account to navigate to the Service account web page.
  2. Scroll to the Keys part and click on the Add Key dropdown and click on on the Create new key choice which opens a modal.
  3. Choose a JSON file format and click on the Create button on the backside proper of the modal.

Observe: It is strongly recommended to maintain a service account key personal and never commit it to any version control system because it incorporates extremely delicate information a few mission on the Google Cloud. This may be finished by including the file to the .gitignore file.

With a service account created and a service account key accessible inside our mission’s listing, we will use the Dialogflow consumer library to ship and obtain information from the Dialogflow agent.

// agentRoute.js
require("dotenv").config();

const categorical = require("categorical")
const Dialogflow = require("@google-cloud/dialogflow")
const { v4 as uuid } = require("uuid")
const Path = require("path")
 
const app = categorical();

app.submit("/text-input", async (req, res) => {
  const { message } = req.physique;

  // Create a brand new session
   const sessionClient = new Dialogflow.SessionsClient({
    keyFilename: Path.be a part of(__dirname, "./key.json"),
  });

  const sessionPath = sessionClient.projectAgentSessionPath(
    course of.env.PROJECT_ID,
    uuid()
  );

  // The dialogflow request object
  const request = {
    session: sessionPath,
    queryInput: {
      textual content: {
        // The question to ship to the dialogflow agent
        textual content: message,
      },
    },
  };

  // Sends information from the agent as a response
  strive {
    const responses = await sessionClient.detectIntent(request);
    res.standing(200).ship({ information: responses });
  } catch (e) {
    console.log(e);
    res.standing(422).ship({ e });
  }
});

module.exports = app;

All the route above sends information to the Dialogflow agent and receives a response via the next steps.

  • First
    It authenticates with the Google cloud then it creates a session with Dialogflow utilizing the projectID of Google cloud mission linked to the Dialogflow agent and likewise a random ID to establish the session created. In our software, we’re making a UUID identifier on every session created utilizing the JavaScript UUID package deal. That is very helpful when logging or tracing all conversations dealt with by a Dialogflow agent.
  • Second
    We create a request object information following the desired format within the Dialogflow documentation. This request object incorporates the created session and the message information gotten from the request physique which is to be handed to the Dialogflow agent.
  • Third
    Utilizing the detectIntent technique from the Dialogflow session, we ship the request object asynchronously and await the Agent’s response utilizing ES6 async / await syntax in a try-catch block ought to the detectIntent technique return an exception, we will catch the error and return it, relatively than crashing your entire software. A pattern of the response object returned from the Agent is supplied within the Dialogflow documentation and will be inspected to know the right way to extract the info from the article.

We are able to make use of Postman to check the Dialogflow connection carried out above within the dialogflow-response route. Postman is a collaboration platform for API growth with options to check APIs in-built both growth or manufacturing levels.

Observe: If not already put in, the Postman desktop software just isn’t wanted to check an API. Ranging from September 2020, Postman’s internet consumer moved right into a Typically Accessible (GA) state and can be utilized immediately from a browser.

Utilizing the Postman Internet Consumer, we will both create a brand new workspace or use an present one to create a POST request to our API endpoint at http://localhost:5000/api/agent/text-input and add information with a key of message and worth of “Hello There” into the question parameters.

On the click on of the Ship button, a POST request will probably be made to the working Categorical server — with a response much like the one proven within the picture beneath:

Testing the text-input API endpoint using Postman.
Testing the text-input API endpoint utilizing Postman. (Large preview)

Inside the picture above, we will see the prettified response information from the Dialogflow agent via the Categorical server. The information returned is formatted based on the sample response definition given within the Dialogflow Webhook documentation.

Dealing with Voice Inputs

By default, all Dialogflow brokers are enabled to course of each textual content and audio information and likewise return a response in both textual content or audio format. Nevertheless, working with audio enter or output information could be a bit extra advanced than textual content information.

To deal with and course of voice inputs, we’ll start the implementation for the /voice-input endpoint that we’ve beforehand created with a view to obtain audio recordsdata and ship them to Dialogflow in change for a response from the Agent:

// agentRoutes.js
import { pipeline, Remodel } from "stream";
import busboy from "connect-busboy";
import util from "promisfy"
import Dialogflow from "@google-cloud/dialogflow"

const app = categorical();

app.use(
  busboy({
    speedy: true,
  })
);

app.submit("/voice-input", (req, res) => {
  const sessionClient = new Dialogflow.SessionsClient({
    keyFilename: Path.be a part of(__dirname, "./recommender-key.json"),
  });
  const sessionPath = sessionClient.projectAgentSessionPath(
    course of.env.PROJECT_ID,
    uuid()
  );

  // remodel right into a promise
  const pump = util.promisify(pipeline);

  const audioRequest = {
    session: sessionPath,
    queryInput: {
      audioConfig: {
        audioEncoding: "AUDIO_ENCODING_OGG_OPUS",
        sampleRateHertz: "16000",
        languageCode: "en-US",
      },
      singleUtterance: true,
    },
  };
  
  const streamData = null;
  const detectStream = sessionClient
    .streamingDetectIntent()
    .on("error", (error) => console.log(error))
    .on("information", (information) => {
      streamData = information.queryResult    
    })
    .on("finish", (information) => {
      res.standing(200).ship({ information : streamData.fulfillmentText }}
    }) 
  
  detectStream.write(audioRequest);

  strive {
    req.busboy.on("file", (_, file, filename) => {
      pump(
        file,
        new Remodel({
          objectMode: true,
          remodel: (obj, _, subsequent) => {
            subsequent(null, { inputAudio: obj });
          },
        }),
        detectStream
      );
    });
  } catch (e) {
    console.log(`error  : ${e}`);
  }
});

At a excessive overview, the /voice-input route above receives a consumer’s voice enter as a file containing the message being spoken to the chat assistant and sends it to the Dialogflow agent. To grasp this course of higher, we will break it down into the next smaller steps:

  • First, we add and use connect-busboy as an Express middleware for parsing type information being despatched within the request from the online software. After which we authenticate with Dialogflow utilizing the Service Key and create a session, the identical means we did within the earlier route.
    Then utilizing the promisify technique from the built-in Node.js util module, we get and save a promise equal of the Stream pipeline technique for use later to pipe a number of streams and likewise carry out a clear up after the streams are accomplished.
  • Subsequent, We create a request object containing the Dialogflow authentication session and a configuration for the audio file about to be despatched to Dialogflow. The nested audio configuration object allows the Dialogflow agent to carry out a Speech-To-Text conversion on the despatched audio file.
  • Subsequent, utilizing the created session and the request object, we detect a consumer’s intent from the audio file utilizing detectStreamingIntent technique which opens up a brand new information stream from the Dialogflow agent to the backend software. Information will ship again in small bits via this stream and utilizing the info “occasion” from the readable stream we retailer the info in streamData variable for later use. After the stream is closed the “finish” occasion is fired and we ship the response from the Dialogflow agent saved within the streamData variable to the Internet Software.
  • Lastly utilizing the file stream occasion from connect-busboy, we obtain the stream of the audio file despatched within the request physique and we additional go it into the promise equal of Pipeline which we created beforehand. The operate of that is to pipe the audio file stream coming in from request to the Dialogflow stream, we pipe the audio file stream to the stream opened by the detectStreamingIntent technique above.

To check and make sure that the steps above are working as laid out, we will make a take a look at request containing an audio file within the request physique to the /voice-input endpoint utilizing Postman.

Testing the voice-input API endpoint using Postman.
Testing the voice-input API endpoint utilizing postman with a recorded voice file. (Large preview)

The Postman consequence above reveals the response gotten after making a POST request with the form-data of a recorded voice word message saying “Hello” included within the request’s physique.

At this level, we now have a useful Categorical.js software that sends and receives information from Dialogflow, the 2 components of this text are finished. The place at the moment are left with integrating this Agent into an internet software by consuming the APIs created right here from a Reactjs software.

Integrating Into A Internet Software

To devour our constructed REST API we’ll develop this present React.js software which already has a house web page displaying a listing of wines fetched from an API and help for decorators utilizing the babel proposal decorators plugin. We are going to refactor it just a little by introducing Mobx for state administration and likewise a brand new function to suggest a wine from a chat element utilizing the added REST API endpoints from the Categorical.js software.

To get began, we start to handle the applying’s state utilizing MobX as we create a Mobx retailer with a number of observable values and a few strategies as actions.

// retailer.js

import Axios from "axios";
import { motion, observable, makeObservable, configure } from "mobx";

const ENDPOINT = course of.env.REACT_APP_DATA_API_URL;

class ApplicationStore {
  constructor() {
    makeObservable(this);
  }

  @observable
  isChatWindowOpen = false;

  @observable
  isLoadingChatMessages = false;

  @observable
  agentMessages = [];

  @motion
  setChatWindow = (state) => {
    this.isChatWindowOpen = state;
  };

  @motion
  handleConversation = (message) => {
     this.isLoadingChatMessages = true;
     this.agentMessages.push({ userMessage: message });

     Axios.submit(`${ENDPOINT}/dialogflow-response`,  "Hello",
     )
      .then((res) => {
        this.agentMessages.push(res.information.information[0].queryResult);
        this.isLoadingChatMessages = false;
      })
      .catch((e) => {
        this.isLoadingChatMessages = false;
        console.log(e);
      });
  };
}

export const retailer = new ApplicationStore();

Above we created a retailer for the chat element function inside the software having the next values:

  • isChatWindowOpen
    The worth saved right here controls the visibility of the chat element the place the messages of Dialogflow are proven.
  • isLoadingChatMessages
    That is used to indicate a loading indicator when a request to fetch a response from the Dialogflow agent is made.
  • agentMessages
    This array shops all responses coming from the requests made to get a response from the Dialogflow agent. The information within the array is later displayed within the element.
  • handleConversation
    This technique adorned as an motion provides information into the agentMessages array. First, it provides the consumer’s message handed in as an argument then makes a request utilizing Axios to the backend software to get a response from Dialogflow. After the request is resolved, it provides the response from the request into the agentMessages array.

Observe: Within the absence of the decorators help in an Software, MobX offers makeObservable which can be utilized within the constructor of the goal retailer class. See an instance here.

With the shop setup, we have to wrap your entire software tree with the MobX Supplier higher-order element ranging from the basis element within the index.js file.

import React from "react";
import { Supplier } from "mobx-react";

import { retailer } from "./state/";
import House from "./pages/house";

operate App() {
  return (
    <Supplier ApplicationStore={retailer}>
      <div className="App">
        <House />
      </div>
    </Supplier>
  );
}

export default App;

Above we wrap the basis App element with MobX Supplier and we go within the beforehand created retailer as one of many Supplier’s values. Now we will proceed to learn from the shop inside elements related to the shop.

Creating A Chat Interface

To show the messages despatched or acquired from the API requests, we want a brand new element with some chat interface displaying the messages listed out. To do that, we create a brand new element to show some hard-coded messages first then later we show messages in an ordered record.

// ./chatComponent.js

import React, { useState } from "react";
import { FiSend, FiX } from "react-icons/fi";
import "../types/chat-window.css";

const heart = {
  show: "flex",
  jusitfyContent: "heart",
  alignItems: "heart",
};

const ChatComponent = (props) => {
  const { closeChatwindow, isOpen } = props;
  const [Message, setMessage] = useState("");

  return (
   <div className="chat-container">
      <div className="chat-head">
        <div type={{ ...heart }}>
          <h5> Zara </h5>
        </div>
        <div type={{ ...heart }} className="hover">
          <FiX onClick={() => closeChatwindow()} />
        </div>
      </div>
      <div className="chat-body">
        <ul className="chat-window">
          <li>
            <div className="chat-card">
              <p>Hello there, welcome to our Agent</p>
            </div>
          </li>
        </ul>
        <hr type={{ background: "#fff" }} />
        <type onSubmit={(e) => {}} className="input-container">
          <enter
            className="enter"
            sort="textual content"
            onChange={(e) => setMessage(e.goal.worth)}
            worth={Message}
            placeholder="Start a dialog with our agent"
          />
          <div className="send-btn-ctn">
            <div className="hover" onClick={() => {}}>
              <FiSend type={{ remodel: "rotate(50deg)" }} />
            </div>
          </div>
        </type>
      </div>
    </div>
  );
};

export default ChatComponent

The element above has the essential HTML markup wanted for a chat software. It has a header displaying the Agent’s identify and an icon for closing the chat window, a message bubble containing a hardcoded textual content in a listing tag, and lastly an enter subject having an onChange occasion handler hooked up to the enter subject to retailer the textual content typed into the element’s native state utilizing React’s useState.

A preview of the Chat Component with a hard coded message from the chat Agent
A preview of the Chat Element with a tough coded message from the chat Agent. (Large preview)

From the picture above, the chat element works because it ought to, displaying a styled chat window having a single chat message and the enter subject on the backside. We nonetheless need the message proven to be precise responses gotten from the API request and never hardcoded textual content.

We transfer ahead to refactor the Chat element, this time connecting and making use of values within the MobX retailer inside the element.

// ./elements/chatComponent.js

import React, { useState, useEffect } from "react";
import { FiSend, FiX } from "react-icons/fi";
import { observer, inject } from "mobx-react";
import { toJS } from "mobx";
import "../types/chat-window.css";

const heart = {
  show: "flex",
  jusitfyContent: "heart",
  alignItems: "heart",
};

const ChatComponent = (props) => {
  const { closeChatwindow, isOpen } = props;
  const [Message, setMessage] = useState("");

  const {
    handleConversation,
    agentMessages,
    isLoadingChatMessages,
  } = props.ApplicationStore;

  useEffect(() => {
    handleConversation();
    return () => handleConversation()
  }, []);

  const information = toJS(agentMessages);
 
  return (
        <div className="chat-container">
          <div className="chat-head">
            <div type={{ ...heart }}>
              <h5> Zara {isLoadingChatMessages && "is typing ..."} </h5>
            </div>
            <div type={{ ...heart }} className="hover">
              <FiX onClick={(_) => closeChatwindow()} />
            </div>
          </div>
          <div className="chat-body">
            <ul className="chat-window">
              {information.map(({ fulfillmentText, userMessage }) => (
                <li>
                  {userMessage && (
                    <div
                      type={{
                        show: "flex",
                        justifyContent: "space-between",
                      }}
                    >
                      <p type={{ opacity: 0 }}> . </p>
                      <div
                        key={userMessage}
                        type={{
                          background: "pink",
                          coloration: "white",
                        }}
                        className="chat-card"
                      >
                        <p>{userMessage}</p>
                      </div>
                    </div>
                  )}
                  {fulfillmentText && (
                    <div
                      type={{
                        show: "flex",
                        justifyContent: "space-between",
                      }}
                    >
                      <div key={fulfillmentText} className="chat-card">
                        <p>{fulfillmentText}</p>
                      </div>
                      <p type={{ opacity: 0 }}> . </p>
                    </div>
                  )}
                </li>
              ))}
            </ul>
            <hr type={{ background: "#fff" }} />
            <type
              onSubmit={(e) => {
                e.preventDefault();
                handleConversation(Message);
              }}
              className="input-container"
            >
              <enter
                className="enter"
                sort="textual content"
                onChange={(e) => setMessage(e.goal.worth)}
                worth={Message}
                placeholder="Start a dialog with our agent"
              />
              <div className="send-btn-ctn">
                <div
                  className="hover"
                  onClick={() => handleConversation(Message)}
                >
                  <FiSend type={{ remodel: "rotate(50deg)" }} />
                </div>
              </div>
            </type>
          </div>
        </div>
     );
};

export default inject("ApplicationStore")(observer(ChatComponent));

From the highlighted components of the code above, we will see that your entire chat element has now been modified to carry out the next new operations;

  • It has entry to the MobX retailer values after injecting the ApplicationStore worth. The element has additionally been made an observer of those retailer values so it re-renders when one of many values modifications.
  • We begin the dialog with the Agent instantly after the chat element is opened by invoking the handleConversation technique inside a useEffect hook to make a request instantly the element is rendered.
  • We at the moment are making use of the isLoadingMessages worth inside the Chat element header. When a request to get a response from the Agent is in flight, we set the isLoadingMessages worth to true and replace the header to Zara is typing…
  • The agentMessages array inside the retailer will get up to date to a proxy by MobX after its values are set. From this element, we convert that proxy again to an array utilizing the toJS utility from MobX and retailer the values in a variable inside the element. That array is additional iterated upon to populate the chat bubbles with the values inside the array utilizing a map operate.

Now utilizing the chat element we will sort in a sentence and anticipate a response to be displayed within the agent.

Chat component showing a list data returned from the HTTP request to the express application.
Chat element displaying a listing information returned from the HTTP request to the categorical software. (Large preview)

Recording Consumer Voice Enter

By default, all Dialogflow brokers can settle for both voice or text-based enter in any specified language from a consumer. Nevertheless, it requires a number of changes from an internet software to achieve entry to a consumer’s microphone and document a voice enter.

To attain this, we modify the MobX retailer to make use of the HTML MediaStream Recording API to document a consumer’s voice inside two new strategies within the MobX retailer.

// retailer.js

import Axios from "axios";
import { motion, observable, makeObservable } from "mobx";

class ApplicationStore {
  constructor() {
    makeObservable(this);
  }

  @observable
  isRecording = false;

  recorder = null;
  recordedBits = [];

  @motion
  startAudioConversation = () => {
    navigator.mediaDevices
      .getUserMedia({
        audio: true,
      })
      .then((stream) => {
        this.isRecording = true;
        this.recorder = new MediaRecorder(stream);
        this.recorder.begin(50);

        this.recorder.ondataavailable = (e) => {
           this.recordedBits.push(e.information);
        };
      })
      .catch((e) => console.log(`error recording : ${e}`));
  };
};

On the click on of the document icon from the chat element, the startAudioConversation technique within the MobX retailer above is invoked to set the tactic the observable isRecording property is to true , for the chat element to offer visible suggestions to indicate a recording is in progress.

Utilizing the browser’s navigator interface, the Media Device object is accessed to request the consumer’s gadget microphone. After permission is granted to the getUserMedia request, it resolves its promise with a MediaStream information which we additional go to the MediaRecorder constructor to create a recorder utilizing the media tracks within the stream returned from the consumer’s gadget microphone. We then retailer the Media recorder occasion within the retailer’s recorder property as we’ll entry it from one other technique in a while.

Subsequent, we name the beginning technique on the recorder occasion, and after the recording session is ended, the ondataavailable operate is fired with an occasion argument containing the recorded stream in a Blob which we retailer within the recordedBits array property.

Logging out the info within the occasion argument handed into the fired ondataavailable occasion, we will see the Blob and its properties within the browser console.

Browser Devtools console showing logged out Blob created by the Media Recorder after a recording is ended.Pull Quotes
Browser Devtools console displaying logged out Blob created by the Media Recorder after a recording is ended. (Large preview)

Now that we will begin a MediaRecorder stream, we want to have the ability to cease the MediaRecorder stream when a consumer is finished recording their voice enter and ship the generated audio file to the Categorical.js software.

The brand new technique added to the shop beneath stops the stream and makes a POST request containing the recorded voice enter.

//retailer.js

import Axios from "axios";
import { motion, observable, makeObservable, configure } from "mobx";

const ENDPOINT = course of.env.REACT_APP_DATA_API_URL;

class ApplicationStore {
  constructor() {
    makeObservable(this);
  }

  @observable
  isRecording = false;

  recorder = null;
  recordedBits = []; 

  @motion
  closeStream = () => {
    this.isRecording = false;
    this.recorder.cease();
    
    this.recorder.onstop = () => {
      if (this.recorder.state === "inactive") {
        const recordBlob = new Blob(this.recordedBits, {
          sort: "audio/mp3",
        });

        const inputFile = new File([recordBlob], "enter.mp3", {
          sort: "audio/mp3",
        });
        const formData = new FormData();
        formData.append("voiceInput", inputFile);

        Axios.submit(`${ENDPOINT}/api/agent/voice-input`, formData, {
          headers: {
            "Content material-Kind": "multipart/formdata",
          },
        })
          .then((information) => {})
          .catch((e) => console.log(`error importing audio file : ${e}`));
      }
    };
  };
}

export const retailer = new ApplicationStore();

The strategy above executes the MediaRecorder’s cease technique to cease an lively stream. Inside the onstop occasion fired after the MediaRecorder is stopped, we create a brand new Blob with a music sort and append it right into a created FormData.

Because the final step., we make POST request with the created Blob added to the request physique and a Content material-Kind: multipart/formdata added to the request’s headers so the file will be parsed by the connect-busboy middleware from the backend-service software.

With the recording being carried out from the MobX retailer, all we have to add to the chat-component is a button to execute the MobX actions to start out and cease the recording of the consumer’s voice and likewise a textual content to indicate when a recording session is lively.

import React from 'react'

const ChatComponent = ({ ApplicationStore }) => {
  const {
     startAudiConversation,
     isRecording,
     handleConversation,
     endAudioConversation,
     isLoadingChatMessages
    } = ApplicationStore

  const [ Message, setMessage ] = useState("") 

    return (
        <div>
           <div className="chat-head">
            <div type={{ ...heart }}>
              <h5> Zara {} {isRecording && "is listening ..."} </h5>
            </div>
            <div type={{ ...heart }} className="hover">
              <FiX onClick={(_) => closeChatwindow()} />
            </div>
          </div>          
   
          <type
              onSubmit={(e) => {
                  e.preventDefault();
                  handleConversation(Message);
                }}
                className="input-container"
              >
                <enter
                  className="enter"
                  sort="textual content"
                  onChange={(e) => setMessage(e.goal.worth)}
                  worth={Message}
                  placeholder="Start a dialog with our agent"
                />
                <div className="send-btn-ctn">
                  {Message.size > 0 ? (
                    <div
                      className="hover"
                      onClick={() => handleConversation(Message)}
                    >
                      <FiSend type={{ remodel: "rotate(50deg)" }} />
                    </div>
                  ) : (
                    <div
                      className="hover"
                      onClick={() =>  handleAudioInput()}
                    >
                      <FiMic />
                    </div>
                  )}
                </div>
              </type>
        </div>     
    )
}

export default ChatComponent

From the highlighted half within the chat element header above, we use the ES6 ternary operators to modify the textual content to “Zara is listening ….” every time a voice enter is being recorded and despatched to the backend software. This offers the consumer suggestions on what’s being finished.

Additionally, apart from the textual content enter, we added a microphone icon to tell the consumer of the textual content and voice enter choices accessible when utilizing the chat assistant. If a consumer decides to make use of the textual content enter, we swap the microphone button to a Ship button by counting the size of the textual content saved and utilizing a ternary operator to make the swap.

We are able to take a look at the newly related chat assistant a few occasions by utilizing each voice and textual content inputs and watch it reply precisely like it will when utilizing the Dialogflow console!

Conclusion

Within the coming years, using language processing chat assistants in public companies may have turn out to be mainstream. This text has supplied a fundamental information on how one among these chat assistants constructed with Dialogflow will be built-in into your individual internet software via using a backend software.

The constructed software has been deployed utilizing Netlify and will be discovered here. Be at liberty to discover the Github repository of the backend categorical software here and the React.js internet software here. They each comprise an in depth README to information you on the recordsdata inside the two tasks.

References

Smashing Editorial
(ks, vf, yk, il)



Source link