Popular in Engineering

Engineering

Spatiotemporal evaporating droplet dynamics on fomites enhances long term bacterial pathogenesis

Indian Institute of Science Saptarshi Basu

Civil Engineering

Suitability of excavated London clay as a supplementary cementitious material

ARUP Fragkoulis Kanavaris

Chemistry

Polyfluorene-Wrapped Carbon Nanotubes Decorated with Photocleavable Side-Chains

McMaster University Alex Adronov

Electronic Engineering

A Trusted and Privacy-Preserving Internet of Mobile Energy

Queensland University of Technology

Engineering

Water Security: Why We Need Global Solutions

Cardiff University

Engineering

Decision-Making Principles for Better Software Design Decisions

Swinburne University of Technology

Civil Engineering

Adaptive Traffic Signal Control With Deep Reinforcement Learning and High Dimensional Sensory Inputs

University of Toronto

Control of a Wheelchair-Mounted Robotic Arm System

View more insights in Professional - Engineering - Electronic Engineering

View more figures from University of South Florida

View the page of Redwan Alqasemi

Read the Study

Redwan Alqasemi discusses how a wheelchair-mounted robotic arm can enhance the manipulation capabilities of individuals with disabilities that are using power wheelchairs, and reduce dependence on human aides.

Image courtesy of interviewee. May 1, 2024

Generating Workspace

Export as PDF

This video video discusses the development of an assistive robotic arm designed to help individuals with physical disabilities, particularly those who use wheelchairs. The motive behind this work is to empower such individuals to perform daily activities, which they often struggle with due to their limitations. The proposed solution is a robotic arm with seven degrees of freedom, mimicking the function of a human arm, allowing users to reach and manipulate objects.

Key aspects include:

1. **Robotic Arm Design**: The arm features seven joints for enhanced manipulation capabilities and can be attached to a power wheelchair. This video video integration allows for combined movement and control.

2. **User Interfaces**: The system offers multiple control interfaces, including a touchscreen, a force-feedback device, a joystick, and a brain-computer interface (BCI). The BCI is particularly aimed at individuals with severe paralysis.

3. **Operation Modes**: Users can engage with the robotic arm via teleoperation (direct control) or autonomously (selecting tasks for the arm to perform automatically).

4. **Learning and Adaptation**: The project envisions a future where the robotic arm will learn from user interactions, adapting to their specific needs and preferences, much like a child learns from their environment.

5. **Challenges**: This video video highlights challenges with interface accuracy, especially using the BCI, which can decline after about 30 minutes due to user fatigue.

6. **Future Directions**: The project discusses ongoing funding and development efforts to enhance the robotic arm's capabilities and further its learning processes for better adaptation to users.

Overall, the initiative aims to enhance the quality of life for people with disabilities by leveraging robotics and innovative user interfaces.

Language:

Export as PDF

X-TIMESTAMP-MAP:LOCAL=00:00:00.000,MPEGTS=900000

00:00.425 --> 00:00.885
My pleasure.

00:00.945 --> 00:02.965
Um, thank you for, you know, for the introduction.

00:03.465 --> 00:07.045
Um, lemme give you some, some motivation behind this work.

00:07.065 --> 00:10.765
And the need, you know, this work is done for people

00:10.765 --> 00:14.365
with disabilities, mostly, physical disabilities,

00:14.425 --> 00:16.085
who are confined to our wheelchair.

00:16.745 --> 00:18.845
And, oftentimes we take things for granted.

00:19.025 --> 00:20.885
You know, if something falls off of your hand,

00:20.885 --> 00:22.965
you just lean down and pick it up, no problems.

00:23.615 --> 00:25.595
But a personal wheelchair, you know,

00:25.595 --> 00:27.875
very likely would need assistance to, you know, to do that.

00:28.885 --> 00:30.625
So, assistive technologies help people

00:30.625 --> 00:32.345
with disabilities do activities of their living

00:32.375 --> 00:34.585
that we all do without having, having any problems.

00:35.405 --> 00:37.985
Um, furthermore, you know, a lot of people would like

00:38.265 --> 00:40.785
to get jobs, but there's always a problem with

00:41.315 --> 00:43.665
doing activities or tasks related to these jobs.

00:43.965 --> 00:46.785
Um, so the thought behind this is that if we can provide

00:47.545 --> 00:50.065
a robotic arm that can be attached to a power wheelchair,

00:50.655 --> 00:53.265
then the arm can be there, their hand, basically,

00:53.365 --> 00:55.425
so they can reach out to different objects,

00:55.485 --> 00:56.825
and manipulate different objects

00:57.205 --> 00:58.945
or do different tasks, whether it related

00:59.005 --> 01:01.625
to daily activities or, or job tasks.

01:02.465 --> 01:04.005
Uh, so that's the motive behind this.

01:04.105 --> 01:08.125
And, the thought we thought about the solution is

01:08.125 --> 01:11.245
that why don't we create the wheelchair mounted robotic arm

01:11.245 --> 01:14.685
where we can combine the mobility of the wheelchair

01:14.685 --> 01:16.925
and the manipulation of the arm into one body.

01:16.925 --> 01:18.845
And I'm gonna talk a little bit more about this later on.

01:19.715 --> 01:20.975
Um, but that's the motive.

01:21.275 --> 01:25.415
Um, and, the solution we created is,

01:25.855 --> 01:28.375
creating a seven degree of freedom robotic arm,

01:28.375 --> 01:30.975
meaning seven joints, which is similar to a human arm.

01:30.975 --> 01:33.295
If you think about your human arm from shoulder to wrist,

01:33.295 --> 01:35.975
you have seven joints, three in the shoulder, one

01:35.975 --> 01:38.135
or two in the elbow, and then one or two in the wrist.

01:38.355 --> 01:40.215
Um, that, or meaning

01:40.215 --> 01:42.575
that one joint is either counted at in the

01:42.575 --> 01:43.735
elbow or in the, in the wrist.

01:44.275 --> 01:45.815
Uh, but in total we have seven joints.

01:46.665 --> 01:49.445
Uh, and as the audience may know,

01:49.465 --> 01:52.525
we need six joints at a minimum to reach any position

01:52.525 --> 01:54.765
and orientation and, and in space, right?

01:55.305 --> 01:58.645
Uh, that, that would take care of the X, Y, z, in space

01:58.705 --> 02:00.245
and the rotation about X, Y, Z.

02:00.245 --> 02:02.565
So that takes care of the position and orientation.

02:03.445 --> 02:06.025
So when we have a seventh joint that gives us that, what,

02:06.025 --> 02:07.945
what we call redundancy, which means

02:07.945 --> 02:10.065
that you can still reach that same position,

02:10.205 --> 02:12.065
but in different configuration of your arm.

02:12.205 --> 02:15.545
That's why if you, if you fix your wrist somewhere

02:15.925 --> 02:19.145
and your elbow is fixed, you can still move your arm,

02:19.215 --> 02:20.905
that one extra degree of freedom, that,

02:20.905 --> 02:21.905
that's why we put redundant.

02:22.755 --> 02:24.895
Uh, and because of that extra degree of freedom, you can,

02:24.955 --> 02:28.795
you have the freedom to do a lot of, to,

02:28.795 --> 02:31.035
to resolve the inverse kinematics of this arm.

02:31.035 --> 02:33.355
You have to do a lot of, optimization.

02:33.775 --> 02:36.315
You can optimize for energy, you know, for, for,

02:36.315 --> 02:38.595
for different things that you can include in your grads.

02:39.475 --> 02:42.335
Um, so, so that's the, that's the solution that we came up

02:42.335 --> 02:44.375
with, attached to our wheelchair.

02:45.205 --> 02:47.265
Uh, and in terms of the interface, we thought

02:47.265 --> 02:48.825
of different ways to interface this.

02:49.005 --> 02:51.625
Um, of course, a lot of people can,

02:51.625 --> 02:53.545
control robotic arms when they are healthy,

02:54.445 --> 02:58.785
but you have, if you have amputations or you have, CP

02:58.785 --> 03:01.205
Or any other diseases that might hinder your,

03:01.205 --> 03:05.325
your arm from controlling, anything,

03:05.325 --> 03:06.805
then you have to, to be creative in

03:06.805 --> 03:08.045
what kind of interfaces you create.

03:08.145 --> 03:11.005
So we came up with, three different,

03:11.025 --> 03:12.245
or four different interfaces.

03:12.245 --> 03:13.325
One of them is a touch screen

03:13.325 --> 03:15.525
where you can just use touch to control the arm.

03:16.365 --> 03:19.025
Uh, another one is what we call the phantom Omni,

03:19.025 --> 03:20.745
which is a, a sty that you hold

03:20.765 --> 03:22.865
and gives you force feedback, haptic feedback,

03:23.245 --> 03:25.785
so you can feel some forces coming back, back to you.

03:27.235 --> 03:28.295
Uh, and then,

03:28.455 --> 03:30.655
a third one is the sixth degree freedom joystick,

03:30.755 --> 03:33.655
six axis joystick, which basically also takes care of,

03:34.155 --> 03:36.695
three position and three orientation in space.

03:37.715 --> 03:40.375
And then the force, which is the most extravagant one,

03:40.475 --> 03:43.255
is the brain machine interface, or brain computer interface.

03:44.005 --> 03:47.345
Uh, and this is made for people with severe paralysis,

03:47.565 --> 03:49.825
or severe like a LS patients, for example,

03:49.825 --> 03:52.145
that have no control whatsoever of,

03:52.145 --> 03:53.225
of their, of their muscles.

03:54.235 --> 03:56.775
Um, they have completely,

03:57.135 --> 03:58.295
functional cognitive abilities,

03:59.195 --> 04:00.455
but they cannot move anything.

04:00.455 --> 04:02.775
They can't even, move their head or,

04:02.995 --> 04:04.495
or, you know, tell you yes or no.

04:05.305 --> 04:07.885
So for that, you know, we can tap into their,

04:08.135 --> 04:10.245
brain signal, the EEG signal coming from the brain.

04:10.625 --> 04:11.765
We can stimulate that signal.

04:11.825 --> 04:13.365
I'm gonna talk a little bit more about this later on,

04:14.045 --> 04:16.565
stimulate the signal and then, have them control the arm

04:16.565 --> 04:17.645
to do their activities available.

04:17.985 --> 04:20.845
Uh, as much as possible, the first thing we needed

04:20.865 --> 04:23.125
to do is design the arm itself, right?

04:23.305 --> 04:25.325
We needed to, to have an arm that's reliable

04:25.505 --> 04:27.565
and something that we can connect to the wheelchair.

04:27.625 --> 04:29.925
And back when we, when we designed this arm,

04:29.935 --> 04:32.565
there were no commercial arms available that can attach

04:32.565 --> 04:33.565
to a power wheelchair.

04:34.415 --> 04:38.835
Um, so, so we designed this arm, using seven joints.

04:38.975 --> 04:42.035
Uh, we used harmonic drives, for, for gear,

04:42.035 --> 04:43.355
for gear, reduction.

04:43.975 --> 04:47.395
Um, and we use also servo motors, to, to do this.

04:47.395 --> 04:50.475
And then we designed the controller in such a way

04:50.475 --> 04:52.795
that you can control, 10 AEs,

04:53.095 --> 04:54.875
10 independent, motors.

04:55.055 --> 04:57.675
Um, seven for the arm, one for the gripper

04:57.675 --> 04:59.555
that we designed also,

04:59.655 --> 05:01.155
and then took two for the wheelchair.

05:01.155 --> 05:02.395
So you can integrate the wheelchair

05:02.395 --> 05:03.475
motion into the arm motion.

05:03.735 --> 05:05.795
Um, and again, I'm, I'm gonna talk about this,

05:05.795 --> 05:06.875
further details later on.

05:07.415 --> 05:11.055
So, so these 10 axis will take care

05:11.055 --> 05:13.535
of controlling all these, these motions.

05:14.115 --> 05:17.175
And then the gripper, we designed it in such a way that we,

05:17.175 --> 05:19.015
we used a lot of commercial grippers, you know,

05:19.075 --> 05:20.735
it wasn't satisfying to our work.

05:20.875 --> 05:22.175
So we designed our own gripper

05:22.175 --> 05:26.135
where you have four different, surfaces on,

05:26.135 --> 05:29.655
on the claws, one cylindrical, one spherical, and one flat,

05:29.675 --> 05:32.735
and then one tiny for, for handling small little, cards

05:32.755 --> 05:36.135
or papers, spherical for spherical objects, cylindrical

05:36.135 --> 05:37.175
for iCal objects.

05:37.635 --> 05:39.975
And we also designed it, designed it to have a passive,

05:40.195 --> 05:42.295
you know, passive joint where it com, you know,

05:42.295 --> 05:44.295
the two claws com, you know, conform

05:44.295 --> 05:45.375
to the shape of the object.

05:45.805 --> 05:48.375
Like if you're holding, for example, a taper object like,

05:48.795 --> 05:51.815
a cup, you know, it conforms and, and,

05:51.815 --> 05:52.855
and I fix it up nicely.

05:53.985 --> 05:56.805
Um, and then with that, of course, you know, once

05:56.805 --> 06:00.285
We design the hardware, we put it together,

06:01.065 --> 06:03.005
we put the car t the car, the control

06:03.025 --> 06:04.205
to be done in two different ways.

06:04.265 --> 06:08.405
Either Cartesian control or joint control, joint control.

06:08.405 --> 06:09.965
We control each joint separately.

06:10.145 --> 06:11.925
Um, so we say joint one and two

06:11.925 --> 06:13.605
and three go this much and, and so forth.

06:14.475 --> 06:16.615
Uh, Cartesian control, we do reverse matic

06:16.615 --> 06:19.335
where we control the end effector or the gripper, right?

06:20.035 --> 06:22.295
And X, Y, Z, and the rotation of X, Y, z in that, in

06:22.295 --> 06:23.855
that case, you do reverse matic to figure out

06:23.855 --> 06:27.015
what are the joint motions needed for each joint to produce

06:27.015 --> 06:29.695
that end trajectory, for, for, for your gripper.

06:30.535 --> 06:32.515
Uh, so that will, that's what we call Cartesian control.

06:33.425 --> 06:34.965
Uh, we design both controllers.

06:35.765 --> 06:38.305
Uh, and then while we were doing some testing,

06:38.525 --> 06:41.585
we found out that, you know, what's the best way

06:41.585 --> 06:42.825
for the user to control this?

06:42.825 --> 06:44.305
There are two different thoughts about this.

06:44.445 --> 06:47.705
Now, if you're sitting in a wheelchair forward might be

06:47.705 --> 06:49.425
forward based on your wheelchair

06:50.085 --> 06:51.785
or forward based on where the gripper is.

06:51.785 --> 06:53.985
This is forward based on the gripper,

06:53.985 --> 06:55.065
if the gripper is going this way.

06:55.565 --> 06:56.665
But then for your wheelchair,

06:56.665 --> 06:57.585
if you're sitting on a wheelchair,

06:57.585 --> 06:58.625
this will be forward, right?

06:59.585 --> 07:00.845
So, so we designed this.

07:00.855 --> 07:03.765
Again, this is, you know, this has been done using mapping.

07:04.025 --> 07:06.005
Uh, we map the motion to be in reference

07:06.005 --> 07:07.605
to the wheelchair coordinate system

07:07.665 --> 07:10.005
or the gripper coordinate system, right?

07:10.975 --> 07:13.355
Uh, just like, for example, a pilot, you know,

07:13.355 --> 07:16.555
the directions for a pilot is based on the aircraft itself.

07:17.175 --> 07:19.875
Uh, but if you're doing a, you know, some kind of a,

07:20.475 --> 07:22.515
a remote controlled, device,

07:22.535 --> 07:24.435
and you're looking physically into the, the,

07:24.435 --> 07:26.355
the helicopter plane, you forward 'em backwards,

07:26.355 --> 07:27.955
are based on the ground, frame.

07:30.125 --> 07:33.465
Um, so, so we did that, control,

07:34.165 --> 07:36.625
and we also did, two different ways

07:36.685 --> 07:37.825
of operating the arm.

07:38.285 --> 07:41.545
One in teleoperation, meaning that you control directions

07:41.545 --> 07:44.665
of the end effector and you use your cognitive

07:44.665 --> 07:45.825
abilities to complete the task.

07:46.955 --> 07:49.855
Uh, and then we did also autonomous operation where

07:50.635 --> 07:52.855
you specify the object that you'd like to handle

07:52.995 --> 07:54.695
and the task that you'd like to,

07:54.695 --> 07:55.895
to perform with that object.

07:56.445 --> 07:58.775
Once you pick these two, then the, the,

07:58.775 --> 08:00.175
the task would be done autonomously.

08:00.555 --> 08:02.015
Of course, we have cameras, we have allow

08:02.015 --> 08:03.455
sensors embedded into the system.

08:04.285 --> 08:06.705
Uh, and that includes, of course, one of them is the BCI,

08:06.725 --> 08:08.065
the brain computer interface.

08:09.385 --> 08:12.405
Um, I can talk a little bit more about BCI if you'd

08:12.405 --> 08:13.725
like, this time.

08:14.225 --> 08:16.485
So a little brief introduction about the BCI.

08:16.485 --> 08:18.845
The BCI is, you know, brain machine interface, you know, it,

08:18.915 --> 08:22.125
it's, uses different technologies to harness your,

08:22.275 --> 08:24.045
your EEG signal from the brain,

08:24.585 --> 08:27.965
and then you can use this in many different, ways,

08:28.265 --> 08:30.325
to interpret what, what it means, right?

08:30.905 --> 08:33.485
So for this particular work, we use what we call P 300.

08:33.825 --> 08:35.205
Uh, that's positive 300.

08:35.355 --> 08:38.925
What it means is that you stimulate your brain, okay?

08:39.155 --> 08:40.565
With visual cues.

08:41.595 --> 08:43.815
Um, actually, you know, there, there's a term for this.

08:43.835 --> 08:46.415
You know, it's, it's called the steady state,

08:46.415 --> 08:48.055
visual potential.

08:48.855 --> 08:50.355
And what, what, what happens is

08:50.355 --> 08:54.835
that we stimulate your brain using, flashing metrics in

08:54.835 --> 08:56.965
Front of you visually, and we ask you to react

08:56.965 --> 08:58.685
to these mattresses or these flashes.

08:59.345 --> 09:01.285
Uh, when you react to them, of course, we,

09:01.345 --> 09:02.725
we divide them into rows

09:02.725 --> 09:06.485
and columns When you react to them,

09:06.685 --> 09:08.685
a potential just spikes up.

09:09.375 --> 09:10.675
And then we pick that peak

09:11.055 --> 09:13.315
and go back 300 milliseconds to find out

09:13.315 --> 09:14.595
what was flashing at that time.

09:15.015 --> 09:17.475
We filter the signal, we figure out, you know,

09:17.735 --> 09:19.315
of course we average the signal as well.

09:19.655 --> 09:23.675
We filter out, all the, the noise out of it,

09:24.135 --> 09:25.555
and then we can find out,

09:26.025 --> 09:28.115
what you are interested in is this particular cell

09:28.115 --> 09:29.675
that you're looking at, and that's

09:29.675 --> 09:31.075
where we go from, from there.

09:32.345 --> 09:36.895
Okay? So, you know, this technology is, you know,

09:36.915 --> 09:40.655
is proven to be really effective in the first 30 minutes

09:41.425 --> 09:44.615
where you really need a good, strong cognitive abilities.

09:45.135 --> 09:46.935
'cause you have to focus and concentrate on the stimulus.

09:47.875 --> 09:51.255
Uh, after 30 minutes, the, you know, the accuracy goes,

09:51.425 --> 09:53.695
drops down quite dramatically,

09:53.695 --> 09:55.375
because your focus is done, right?

09:56.485 --> 09:58.185
So, so we use this,

09:58.285 --> 10:00.985
and what we did with the interface here is we had the camera

10:00.985 --> 10:03.065
looking into the environment,

10:03.485 --> 10:06.065
and then we take the image, that the camera is,

10:06.065 --> 10:10.385
is seeing, of course, you know, we do point cloud,

10:10.505 --> 10:12.265
data as well to, to get,

10:12.585 --> 10:13.785
whatever objects we need out of it.

10:14.125 --> 10:17.585
We use edge detection to, break,

10:17.585 --> 10:18.585
the image into cells.

10:19.365 --> 10:22.425
Um, and then these cells have seed point in each one

10:22.425 --> 10:24.765
of these cells, that will be flashing.

10:25.265 --> 10:27.145
So if you're interested in, in that image

10:27.145 --> 10:29.985
that you're looking at, if you're interested in picking up

10:29.985 --> 10:33.185
the cup, for example, okay, you look at that image,

10:33.725 --> 10:36.625
you focus on the, on the seed point, that's on the cup.

10:37.325 --> 10:39.585
And then once we figure out this is the object

10:39.585 --> 10:41.825
that you're interested in, we take this

10:41.825 --> 10:43.265
and we take it through object recognition

10:43.265 --> 10:44.345
to find out what this object is.

10:44.895 --> 10:48.105
Once we know what the object is, then we can display a menu

10:48.105 --> 10:49.545
of things that you can do with this object.

10:49.895 --> 10:52.545
Like, if you select a cup, I can fill it,

10:52.625 --> 10:54.385
I can drink from it, I can throw it away,

10:54.425 --> 10:55.465
I can put it work, right?

10:56.285 --> 11:00.175
And then the menu of different tasks start flashing again.

11:00.235 --> 11:03.175
You focus, you, you know, you, you react to the,

11:03.175 --> 11:06.535
to the metrics that's flashing, based on the task

11:06.535 --> 11:09.615
that's on, on display, and choose the task that you'd like.

11:09.965 --> 11:11.095
Once you select the task,

11:11.565 --> 11:12.975
then you have the object on the task,

11:12.975 --> 11:16.245
and then the everything else is done autonomously, right?

11:16.865 --> 11:19.085
So this is doing autonomous motion with the BCI.

11:19.085 --> 11:20.925
If we're doing teleoperation with the BCI,

11:21.425 --> 11:24.125
the flashing matrix will show directions of

11:24.125 --> 11:26.605
where you want the grouper to go, right, left up,

11:26.605 --> 11:27.965
down rotation, whatever, right?

11:28.065 --> 11:29.205
All these six degrees of freedom.

11:29.915 --> 11:32.855
So in this case, you'll have to control the arm motion

11:33.555 --> 11:35.775
and tele operation using the PCI and, and,

11:35.775 --> 11:37.695
and the thoughts of which direction you want to go.

11:38.775 --> 11:40.795
Um, we had pretty good success in this,

11:40.935 --> 11:42.635
and I'm, I'll talk about the results later on.

11:42.635 --> 11:45.995
But you know, it was quite fascinating when you see that

11:47.075 --> 11:51.395
people with almost no way of communication,

11:51.855 --> 11:53.805
can now do some activities that

11:53.835 --> 11:56.565
otherwise they would never be able to do it themselves

11:56.625 --> 11:59.205
or communicate to someone to do it for them, right?

11:59.265 --> 12:01.805
So this way, now they're in total control.

12:02.595 --> 12:07.255
Um, and mind you, you know, if this is the hardest way

12:07.255 --> 12:09.695
to control the arm using the PCI, because again,

12:09.745 --> 12:13.265
after 30, 30 minutes, it becomes random signal.

12:13.445 --> 12:15.465
Uh, there's no, there's no focus anymore.

12:16.665 --> 12:18.325
So if the patient is willing or,

12:18.345 --> 12:22.525
or able to use any different technologies like twitching or,

12:22.585 --> 12:24.365
or blinking or some something like this

12:24.595 --> 12:27.005
that can last a long, a lot longer,

12:27.005 --> 12:29.725
but people who can even, you know, blink,

12:30.315 --> 12:31.805
this could be the, the only solution.

12:32.035 --> 12:33.285
Well, here's the thing.

12:33.315 --> 12:36.605
When the, the first trial we did, we did trials with,

12:36.605 --> 12:39.805
teleoperation, so doing directions and right and left,

12:39.805 --> 12:42.285
and remember we combined the, the arm with the wheelchair.

12:42.305 --> 12:43.885
So if, if you're trying to reach an object

12:43.885 --> 12:44.885
that's far away from you, the

12:44.885 --> 12:45.885
wheelchair will move along, right?

12:46.975 --> 12:50.035
Um, now the downside about this that we found out is

12:50.035 --> 12:53.195
that every command you give from your brain to the,

12:53.195 --> 12:56.945
to the machine takes about 30 seconds, right?

12:57.365 --> 12:59.945
So let's say you are moving, you are commanding the arm

12:59.945 --> 13:01.185
to move, and the wheelchair's moving,

13:01.285 --> 13:02.545
and then suddenly you find the cliff

13:02.605 --> 13:05.265
and you want it to stop By the time you say, Hey, you know,

13:05.265 --> 13:08.105
I want to stop, and you give the stimulation to, to get the,

13:08.125 --> 13:10.985
the, the, the command going, through the, the interface,

13:11.605 --> 13:14.025
30 seconds might be a little too late, right?

13:14.325 --> 13:16.305
So that was our first attempt on this.

13:16.325 --> 13:18.505
And then afterwards we moved into the autonomous,

13:18.505 --> 13:19.585
which is way more efficient

13:19.585 --> 13:21.865
and way more safe, where you don't have

13:21.865 --> 13:22.945
to worry about any of these things.

13:22.975 --> 13:24.785
Sensors can take all of these things.

13:25.565 --> 13:27.345
All you have to do is choose the object

13:27.485 --> 13:29.345
and choose the task for that object,

13:29.845 --> 13:31.585
and things will be done in a safe manner.

13:31.995 --> 13:35.385
Autonomous, I don't think this is included in the paper

13:35.385 --> 13:37.385
that you looked at, but we have a lot of other publications,

13:37.605 --> 13:40.625
following that paper that you looked at,

13:40.625 --> 13:43.105
that shows this, this new method that we, we have adopted.

13:43.725 --> 13:46.065
The thing that we, we, we looked at is that

13:46.605 --> 13:50.515
if the person is on a wheelchair, oftentimes the,

13:50.515 --> 13:53.435
the arm may not be able to reach the target destination

13:53.455 --> 13:54.515
or the target object.

13:54.815 --> 13:58.315
So we adopted what we called,

13:59.935 --> 14:03.475
embedded jacobian, the, the jacobian level.

14:03.475 --> 14:05.795
We combined the jacobian of the wheelchair motion,

14:05.975 --> 14:10.475
the jacobian of the arm manipulation into one jacobian.

14:10.855 --> 14:15.005
And that jacobian, has dimensions of nine by six,

14:15.055 --> 14:17.605
which means that you can control all six degrees

14:17.605 --> 14:20.685
of freedom at the end using the nine joints that you have

14:20.685 --> 14:23.565
between the wheelchair and the arm, right?

14:24.105 --> 14:26.125
Uh, this way it'll give you a lot more leverage on

14:26.125 --> 14:29.645
where you can reach in the, the space surrounding it.

14:30.775 --> 14:33.715
So, we did some experiments on this and some testing,

14:33.935 --> 14:37.895
and, what we found out is that there is a lot

14:37.895 --> 14:40.055
of trade offs that we have to choose from.

14:41.095 --> 14:43.715
Um, first of all, accuracy. Lemme talk about accuracy.

14:43.815 --> 14:46.635
You know, I talked about the flashing mattresses that,

14:46.815 --> 14:48.355
you know, we stimulate the brain with,

14:48.985 --> 14:50.725
and we found a very strong relationship

14:50.725 --> 14:52.965
Between the flashing sequence

14:52.965 --> 14:55.805
because we do it in sequences, so a sequence of flashes,

14:55.805 --> 14:57.645
and then it stops, and then we do another sequence

14:57.645 --> 14:59.245
and then stop, and then another sequence,

14:59.245 --> 15:01.005
and then we average all these sequences, right?

15:01.765 --> 15:04.865
So we found out that the more flash sequences we put,

15:05.005 --> 15:06.985
the more accuracy we get, right?

15:06.985 --> 15:08.545
Because the averaging now, you know, it,

15:08.545 --> 15:12.425
it weeds out all the inaccuracies out of the, the,

15:12.685 --> 15:14.345
the brain signal, that we're getting.

15:14.925 --> 15:17.145
So that was a good relationship that we have published

15:17.145 --> 15:19.505
where, you know, the flashing sequence, you have

15:19.505 --> 15:20.785
to find the sweet spot

15:20.785 --> 15:22.745
where you don't have too many flashes,

15:23.005 --> 15:26.195
and you don't have to have, you know, small amount

15:26.195 --> 15:27.755
of flash flashing sequences

15:27.875 --> 15:29.795
where it becomes very inaccurate.

15:30.545 --> 15:32.445
Uh, if you have too many, that would be too many.

15:32.445 --> 15:33.885
That gets the entire too quickly.

15:34.265 --> 15:36.005
If you have, you know, not enough,

15:36.005 --> 15:37.445
then you'll have less accuracy.

15:37.905 --> 15:39.205
So, so that relationship was,

15:39.305 --> 15:40.765
was one of the results that we found.

15:41.385 --> 15:43.565
Uh, we also found out that the speed of flashing is,

15:43.625 --> 15:45.405
is making a huge difference as well.

15:45.905 --> 15:48.835
Uh, when you flash fast,

15:48.835 --> 15:50.515
things can become like less accurate.

15:50.705 --> 15:54.635
When you flash slower, things become, more accurate,

15:54.635 --> 15:57.275
but it takes more time, for, for this to,

15:57.295 --> 15:58.395
you know, to, to go through.

15:59.285 --> 16:01.905
So that was an interesting, aspect of, of this as well.

16:02.645 --> 16:04.865
Um, accuracy was a hundred percent

16:05.705 --> 16:08.475
when the patient first starts after training, of course,

16:08.715 --> 16:10.075
whenever we use the BCI, you have

16:10.075 --> 16:11.955
to train the system on your brain.

16:12.875 --> 16:15.525
Each individual brain has has its own characteristics.

16:15.585 --> 16:18.845
So before any individual starts with the testing, the,

16:18.845 --> 16:21.125
the system has to be trained on their own,

16:21.385 --> 16:22.525
brain characteristics.

16:23.025 --> 16:25.605
So after the training, we found out the, the,

16:25.745 --> 16:27.615
the very first, you know,

16:27.615 --> 16:29.655
time when you start using the system,

16:30.615 --> 16:32.935
accuracy is a hundred percent for a little while,

16:32.955 --> 16:34.175
for maybe the first five minutes,

16:34.885 --> 16:36.225
and then it starts decaying.

16:36.805 --> 16:38.785
By the time you reach 30 minutes into using this,

16:39.285 --> 16:40.545
you completely lose focus,

16:40.605 --> 16:42.825
and then the signal becomes just a random signal,

16:43.045 --> 16:44.705
or the stimulation does not work anymore.

16:45.845 --> 16:48.665
So, so this is one of the findings that we recommended that,

16:48.725 --> 16:50.945
you know, if this has to be used, it needs

16:50.945 --> 16:54.065
to be used no more than 30 minutes, to get accurate,

16:54.245 --> 16:55.865
you know, accurate information of the brain.

16:56.445 --> 16:58.225
You know, imagine if you're, if you're asked

16:58.225 --> 17:00.225
to look at something and focus on it

17:00.285 --> 17:02.665
and react to a flashing matrix, how,

17:02.925 --> 17:04.825
how much time can you focus on this

17:05.405 --> 17:07.145
and do the reaction that you're supposed

17:07.145 --> 17:08.585
to do without distractions?

17:09.305 --> 17:11.315
It's not easy. It's not really easy.

17:11.735 --> 17:14.795
So, so the human brain, brain can focus only for too long

17:14.795 --> 17:15.955
before they get distracted

17:15.955 --> 17:17.355
and become, you know, you know, fatigue.

17:18.085 --> 17:19.905
Um, and that's human nature.

17:20.165 --> 17:23.665
Um, you know, and and average about 30, 30, 30 minutes into

17:23.715 --> 17:26.465
using the BCI, you start reaching saturation,

17:26.465 --> 17:28.145
and you cannot, you cannot focus anymore.

17:28.575 --> 17:32.425
Well, first of all, you know, of course, one

17:32.425 --> 17:35.905
of the very good things that you can do for people with

17:36.465 --> 17:38.705
physical disabilities is provide assistive technologies.

17:38.965 --> 17:42.145
And, not only they can use it for activities

17:42.145 --> 17:45.225
of daily living, but can, they can use it in finding jobs

17:45.445 --> 17:47.825
and retaining jobs, for that matter,

17:47.825 --> 17:48.825
because independence and

17:48.825 --> 17:50.125
The quality of life is important.

17:50.875 --> 17:52.575
Uh, nobody would like to be dependent on others.

17:53.155 --> 17:56.775
So this technology can provide some, some sort of feeling

17:56.835 --> 17:59.295
of independence and source of income

17:59.295 --> 18:01.255
and source of, you know, being able

18:01.255 --> 18:02.895
to do things yourself, right?

18:03.595 --> 18:08.215
Uh, the BCI is a great way to communicate with people with

18:09.275 --> 18:12.445
a LS it's not the best way to communicate with everybody,

18:12.545 --> 18:14.885
but people who are able to communicate

18:14.985 --> 18:18.205
and do, controls using different means

18:18.785 --> 18:20.125
is always better and recommended.

18:20.265 --> 18:23.085
But last resort, people who cannot use any other interface,

18:24.045 --> 18:28.405
BCI, is a perfect, you know, a place to, to start,

18:28.405 --> 18:31.895
where you can really harness the brain power to

18:32.595 --> 18:34.415
at least communicate to people around you

18:34.415 --> 18:36.575
and do what you want to do on your own

18:36.755 --> 18:39.015
or without having to, to have someone do it for you.

18:40.475 --> 18:43.455
Um, so you know where we're going with this.

18:43.755 --> 18:47.335
Um, right now we have a new funding. Um, it's not new.

18:47.335 --> 18:49.375
It's been, here for a while from National Science

18:49.375 --> 18:52.255
Foundation where we're taking this now to the next level

18:52.905 --> 18:55.855
where we start with a robotic arm as a baby.

18:56.655 --> 18:58.115
Uh, the baby starts in life

18:58.115 --> 18:59.755
knowing absolutely nothing, right?

18:59.975 --> 19:01.915
And as they watch their parents do things,

19:01.915 --> 19:04.235
they start learning how to walk, learning how to eat,

19:04.475 --> 19:05.555
learning how to do things, you know.

19:06.415 --> 19:10.075
So we are doing a system where, you know, you can start

19:10.075 --> 19:12.515
with basic activities that are programmed to it.

19:13.175 --> 19:15.475
Uh, but then as you operate this,

19:15.475 --> 19:17.395
whether it's in Teleoperation,

19:17.775 --> 19:19.835
so any task was divided into subtask.

19:20.375 --> 19:22.355
Um, if the task is new,

19:23.215 --> 19:25.515
the person does the teleoperation and it'll learn.

19:26.095 --> 19:27.155
It doesn't learn the moves.

19:27.255 --> 19:30.755
It learns, of course, from the computer vision, of, of

19:30.755 --> 19:32.115
what it's seeing in the environment,

19:32.345 --> 19:34.915
what it's seeing in the objects that it's handling,

19:35.295 --> 19:37.235
and trying to infer what kind of task you're,

19:37.235 --> 19:38.355
you're making or you're doing.

19:39.215 --> 19:42.395
Um, and then put all of these things in a database where

19:42.935 --> 19:45.635
if you recall it next time, it would remember

19:45.635 --> 19:47.395
what you were doing and what objects you're

19:47.555 --> 19:48.995
handling and so forth.

19:49.895 --> 19:52.595
Uh, so, so after learning, you know, the system,

19:52.625 --> 19:55.115
once they have enough, confidence level

19:55.585 --> 19:57.995
that I can do this sub-task autonomously,

19:58.085 --> 20:00.755
it'll offer the user, Hey, I can do this part.

20:00.755 --> 20:03.115
Like, if you're, if you're trying to, to make an omelet,

20:03.665 --> 20:05.595
okay, can tell you, Hey, you know,

20:05.795 --> 20:07.555
I can really get the extra treatment for you.

20:07.635 --> 20:09.875
I already learned that from you past few times.

20:10.645 --> 20:12.745
So no, go ahead, do it. Okay?

20:13.085 --> 20:15.385
But then as you progress with this, it can say, Hey,

20:15.425 --> 20:16.585
I can do the whole meal for you.

20:17.365 --> 20:18.425
You know, let me do it for you.

20:18.765 --> 20:20.705
So, so as a baby starts learning,

20:20.725 --> 20:21.905
the robot also starts learning

20:21.925 --> 20:24.225
and then start adapting to the person

20:24.245 --> 20:25.505
that's using that system.

20:25.765 --> 20:27.185
Uh, like if you're using the system

20:27.185 --> 20:28.545
and someone else using the system, they,

20:28.545 --> 20:29.585
they learn different things

20:29.925 --> 20:31.185
to go, to be done different ways.

20:32.235 --> 20:33.335
So that's where we're going and

20:33.335 --> 20:34.375
heading with this, project.

20:35.155 --> 20:38.255
Um, again, interface in that case is not really the focus,

20:38.365 --> 20:40.735
it's the focus of, of the learning process

20:40.835 --> 20:42.135
and how you can handle different

20:42.135 --> 20:43.495
objects, without failure.

20:44.275 --> 20:47.455
Um, the interface of course, can also be done the same way,

20:47.455 --> 20:49.735
but that we did it in the previous, previous one.

00:00 - My pleasure. thank you for, you know, for the introduction. lemme give you some, some motivation behind this work. And the need, you know, this work is done for people with disabilities, mostly, physical disabilities, who are confined to our wheelchair.

00:16 - And, oftentimes we take things for granted. You know, if something falls off of your hand, you just lean down and pick it up, no problems. But a personal wheelchair, you know, very likely would need assistance to, you know, to do that. So, assistive technologies help people with disabilities do activities of their living that we all do without having, having any problems. furthermore, you know, a lot of people would like to get jobs, but there's always a problem with

00:41 - doing activities or tasks related to these jobs. so the thought behind this is that if we can provide a robotic arm that can be attached to a power wheelchair, then the arm can be there, their hand, basically, so they can reach out to different objects, and manipulate different objects or do different tasks, whether it related to daily activities or, or job tasks. so that's the motive behind this. And, the thought we thought about the solution is

01:08 - that why don't we create the wheelchair mounted robotic arm where we can combine the mobility of the wheelchair and the manipulation of the arm into one body. And I'm gonna talk a little bit more about this later on. but that's the motive. and, the solution we created is, creating a seven degree of freedom robotic arm, meaning seven joints, which is similar to a human arm. If you think about your human arm from shoulder to wrist, you have seven joints, three in the shoulder, one

01:35 - or two in the elbow, and then one or two in the wrist. that, or meaning that one joint is either counted at in the elbow or in the, in the wrist. but in total we have seven joints. and as the audience may know, we need six joints at a minimum to reach any position and orientation and, and in space, right? that, that would take care of the X, Y, z, in space and the rotation about X, Y, Z.

02:00 - So that takes care of the position and orientation. So when we have a seventh joint that gives us that, what, what we call redundancy, which means that you can still reach that same position, but in different configuration of your arm. That's why if you, if you fix your wrist somewhere and your elbow is fixed, you can still move your arm, that one extra degree of freedom, that, that's why we put redundant. and because of that extra degree of freedom, you can,

02:24 - you have the freedom to do a lot of, to, to resolve the inverse kinematics of this arm. You have to do a lot of, optimization. You can optimize for energy, you know, for, for, for different things that you can include in your grads. so, so that's the, that's the solution that we came up with, attached to our wheelchair. and in terms of the interface, we thought of different ways to interface this. of course, a lot of people can,

02:51 - control robotic arms when they are healthy, but you have, if you have amputations or you have, CP Or any other diseases that might hinder your, your arm from controlling, anything, then you have to, to be creative in what kind of interfaces you create. So we came up with, three different, or four different interfaces. One of them is a touch screen where you can just use touch to control the arm.

03:16 - another one is what we call the phantom Omni, which is a, a sty that you hold and gives you force feedback, haptic feedback, so you can feel some forces coming back, back to you. and then, a third one is the sixth degree freedom joystick, six axis joystick, which basically also takes care of, three position and three orientation in space. And then the force, which is the most extravagant one, is the brain machine interface, or brain computer interface.

03:44 - and this is made for people with severe paralysis, or severe like a LS patients, for example, that have no control whatsoever of, of their, of their muscles. they have completely, functional cognitive abilities, but they cannot move anything. They can't even, move their head or, or, you know, tell you yes or no. So for that, you know, we can tap into their,

04:08 - brain signal, the EEG signal coming from the brain. We can stimulate that signal. I'm gonna talk a little bit more about this later on, stimulate the signal and then, have them control the arm to do their activities available. as much as possible, the first thing we needed to do is design the arm itself, right? We needed to, to have an arm that's reliable and something that we can connect to the wheelchair. And back when we, when we designed this arm,

04:29 - there were no commercial arms available that can attach to a power wheelchair. so, so we designed this arm, using seven joints. we used harmonic drives, for, for gear, for gear, reduction. and we use also servo motors, to, to do this. And then we designed the controller in such a way that you can control, 10 AEs, 10 independent, motors. seven for the arm, one for the gripper

04:57 - that we designed also, and then took two for the wheelchair. So you can integrate the wheelchair motion into the arm motion. and again, I'm, I'm gonna talk about this, further details later on. So, so these 10 axis will take care of controlling all these, these motions. And then the gripper, we designed it in such a way that we, we used a lot of commercial grippers, you know,

05:19 - it wasn't satisfying to our work. So we designed our own gripper where you have four different, surfaces on, on the claws, one cylindrical, one spherical, and one flat, and then one tiny for, for handling small little, cards or papers, spherical for spherical objects, cylindrical for iCal objects. And we also designed it, designed it to have a passive, you know, passive joint where it com, you know, the two claws com, you know, conform

05:44 - to the shape of the object. Like if you're holding, for example, a taper object like, a cup, you know, it conforms and, and, and I fix it up nicely. and then with that, of course, you know, once We design the hardware, we put it together, we put the car t the car, the control to be done in two different ways. Either Cartesian control or joint control, joint control. We control each joint separately.

06:10 - so we say joint one and two and three go this much and, and so forth. Cartesian control, we do reverse matic where we control the end effector or the gripper, right? And X, Y, Z, and the rotation of X, Y, z in that, in that case, you do reverse matic to figure out what are the joint motions needed for each joint to produce that end trajectory, for, for, for your gripper. so that will, that's what we call Cartesian control. we design both controllers.

06:35 - and then while we were doing some testing, we found out that, you know, what's the best way for the user to control this? There are two different thoughts about this. Now, if you're sitting in a wheelchair forward might be forward based on your wheelchair or forward based on where the gripper is. This is forward based on the gripper, if the gripper is going this way. But then for your wheelchair,

06:56 - if you're sitting on a wheelchair, this will be forward, right? So, so we designed this. Again, this is, you know, this has been done using mapping. we map the motion to be in reference to the wheelchair coordinate system or the gripper coordinate system, right? just like, for example, a pilot, you know, the directions for a pilot is based on the aircraft itself. but if you're doing a, you know, some kind of a,

07:20 - a remote controlled, device, and you're looking physically into the, the, the helicopter plane, you forward 'em backwards, are based on the ground, frame. so, so we did that, control, and we also did, two different ways of operating the arm. One in teleoperation, meaning that you control directions of the end effector and you use your cognitive abilities to complete the task.

07:46 - and then we did also autonomous operation where you specify the object that you'd like to handle and the task that you'd like to, to perform with that object. Once you pick these two, then the, the, the task would be done autonomously. Of course, we have cameras, we have allow sensors embedded into the system. and that includes, of course, one of them is the BCI, the brain computer interface.

08:09 - I can talk a little bit more about BCI if you'd like, this time. So a little brief introduction about the BCI. The BCI is, you know, brain machine interface, you know, it, it's, uses different technologies to harness your, your EEG signal from the brain, and then you can use this in many different, ways, to interpret what, what it means, right? So for this particular work, we use what we call P 300. that's positive 300.

08:35 - What it means is that you stimulate your brain, okay? With visual cues. actually, you know, there, there's a term for this. You know, it's, it's called the steady state, visual potential. And what, what, what happens is that we stimulate your brain using, flashing metrics in Front of you visually, and we ask you to react to these mattresses or these flashes. when you react to them, of course, we,

09:01 - we divide them into rows and columns When you react to them, a potential just spikes up. And then we pick that peak and go back 300 milliseconds to find out what was flashing at that time. We filter the signal, we figure out, you know, of course we average the signal as well. We filter out, all the, the noise out of it, and then we can find out,

09:26 - what you are interested in is this particular cell that you're looking at, and that's where we go from, from there. Okay? So, you know, this technology is, you know, is proven to be really effective in the first 30 minutes where you really need a good, strong cognitive abilities. 'cause you have to focus and concentrate on the stimulus. after 30 minutes, the, you know, the accuracy goes, drops down quite dramatically, because your focus is done, right?

09:56 - So, so we use this, and what we did with the interface here is we had the camera looking into the environment, and then we take the image, that the camera is, is seeing, of course, you know, we do point cloud, data as well to, to get, whatever objects we need out of it. We use edge detection to, break, the image into cells. and then these cells have seed point in each one

10:22 - of these cells, that will be flashing. So if you're interested in, in that image that you're looking at, if you're interested in picking up the cup, for example, okay, you look at that image, you focus on the, on the seed point, that's on the cup. And then once we figure out this is the object that you're interested in, we take this and we take it through object recognition to find out what this object is. Once we know what the object is, then we can display a menu

10:48 - of things that you can do with this object. Like, if you select a cup, I can fill it, I can drink from it, I can throw it away, I can put it work, right? And then the menu of different tasks start flashing again. You focus, you, you know, you, you react to the, to the metrics that's flashing, based on the task that's on, on display, and choose the task that you'd like. Once you select the task, then you have the object on the task,

11:12 - and then the everything else is done autonomously, right? So this is doing autonomous motion with the BCI. If we're doing teleoperation with the BCI, the flashing matrix will show directions of where you want the grouper to go, right, left up, down rotation, whatever, right? All these six degrees of freedom. So in this case, you'll have to control the arm motion and tele operation using the PCI and, and, and the thoughts of which direction you want to go.

11:38 - we had pretty good success in this, and I'm, I'll talk about the results later on. But you know, it was quite fascinating when you see that people with almost no way of communication, can now do some activities that otherwise they would never be able to do it themselves or communicate to someone to do it for them, right? So this way, now they're in total control. and mind you, you know, if this is the hardest way to control the arm using the PCI, because again,

12:09 - after 30, 30 minutes, it becomes random signal. there's no, there's no focus anymore. So if the patient is willing or, or able to use any different technologies like twitching or, or blinking or some something like this that can last a long, a lot longer, but people who can even, you know, blink, this could be the, the only solution. Well, here's the thing. When the, the first trial we did, we did trials with,

12:36 - teleoperation, so doing directions and right and left, and remember we combined the, the arm with the wheelchair. So if, if you're trying to reach an object that's far away from you, the wheelchair will move along, right? now the downside about this that we found out is that every command you give from your brain to the, to the machine takes about 30 seconds, right? So let's say you are moving, you are commanding the arm to move, and the wheelchair's moving,

13:01 - and then suddenly you find the cliff and you want it to stop By the time you say, Hey, you know, I want to stop, and you give the stimulation to, to get the, the, the, the command going, through the, the interface, 30 seconds might be a little too late, right? So that was our first attempt on this. And then afterwards we moved into the autonomous, which is way more efficient and way more safe, where you don't have to worry about any of these things.

13:22 - Sensors can take all of these things. All you have to do is choose the object and choose the task for that object, and things will be done in a safe manner. Autonomous, I don't think this is included in the paper that you looked at, but we have a lot of other publications, following that paper that you looked at, that shows this, this new method that we, we have adopted. The thing that we, we, we looked at is that if the person is on a wheelchair, oftentimes the,

13:50 - the arm may not be able to reach the target destination or the target object. So we adopted what we called, embedded jacobian, the, the jacobian level. We combined the jacobian of the wheelchair motion, the jacobian of the arm manipulation into one jacobian. And that jacobian, has dimensions of nine by six, which means that you can control all six degrees of freedom at the end using the nine joints that you have between the wheelchair and the arm, right?

14:24 - this way it'll give you a lot more leverage on where you can reach in the, the space surrounding it. So, we did some experiments on this and some testing, and, what we found out is that there is a lot of trade offs that we have to choose from. first of all, accuracy. Lemme talk about accuracy. You know, I talked about the flashing mattresses that, you know, we stimulate the brain with, and we found a very strong relationship Between the flashing sequence

14:52 - because we do it in sequences, so a sequence of flashes, and then it stops, and then we do another sequence and then stop, and then another sequence, and then we average all these sequences, right? So we found out that the more flash sequences we put, the more accuracy we get, right? Because the averaging now, you know, it, it weeds out all the inaccuracies out of the, the, the brain signal, that we're getting. So that was a good relationship that we have published

15:17 - where, you know, the flashing sequence, you have to find the sweet spot where you don't have too many flashes, and you don't have to have, you know, small amount of flash flashing sequences where it becomes very inaccurate. if you have too many, that would be too many. That gets the entire too quickly. If you have, you know, not enough, then you'll have less accuracy.

15:37 - So, so that relationship was, was one of the results that we found. we also found out that the speed of flashing is, is making a huge difference as well. when you flash fast, things can become like less accurate. When you flash slower, things become, more accurate, but it takes more time, for, for this to, you know, to, to go through. So that was an interesting, aspect of, of this as well.

16:02 - accuracy was a hundred percent when the patient first starts after training, of course, whenever we use the BCI, you have to train the system on your brain. Each individual brain has has its own characteristics. So before any individual starts with the testing, the, the system has to be trained on their own, brain characteristics. So after the training, we found out the, the, the very first, you know,

16:27 - time when you start using the system, accuracy is a hundred percent for a little while, for maybe the first five minutes, and then it starts decaying. By the time you reach 30 minutes into using this, you completely lose focus, and then the signal becomes just a random signal, or the stimulation does not work anymore. So, so this is one of the findings that we recommended that, you know, if this has to be used, it needs

16:50 - to be used no more than 30 minutes, to get accurate, you know, accurate information of the brain. You know, imagine if you're, if you're asked to look at something and focus on it and react to a flashing matrix, how, how much time can you focus on this and do the reaction that you're supposed to do without distractions? It's not easy. It's not really easy. So, so the human brain, brain can focus only for too long

17:14 - before they get distracted and become, you know, you know, fatigue. and that's human nature. you know, and and average about 30, 30, 30 minutes into using the BCI, you start reaching saturation, and you cannot, you cannot focus anymore. Well, first of all, you know, of course, one of the very good things that you can do for people with physical disabilities is provide assistive technologies. And, not only they can use it for activities

17:42 - of daily living, but can, they can use it in finding jobs and retaining jobs, for that matter, because independence and The quality of life is important. nobody would like to be dependent on others. So this technology can provide some, some sort of feeling of independence and source of income and source of, you know, being able to do things yourself, right? the BCI is a great way to communicate with people with

18:09 - a LS it's not the best way to communicate with everybody, but people who are able to communicate and do, controls using different means is always better and recommended. But last resort, people who cannot use any other interface, BCI, is a perfect, you know, a place to, to start, where you can really harness the brain power to at least communicate to people around you and do what you want to do on your own or without having to, to have someone do it for you.

18:40 - so you know where we're going with this. right now we have a new funding. it's not new. It's been, here for a while from National Science Foundation where we're taking this now to the next level where we start with a robotic arm as a baby. the baby starts in life knowing absolutely nothing, right? And as they watch their parents do things, they start learning how to walk, learning how to eat, learning how to do things, you know.

19:06 - So we are doing a system where, you know, you can start with basic activities that are programmed to it. but then as you operate this, whether it's in Teleoperation, so any task was divided into subtask. if the task is new, the person does the teleoperation and it'll learn. It doesn't learn the moves. It learns, of course, from the computer vision, of, of what it's seeing in the environment,

19:32 - what it's seeing in the objects that it's handling, and trying to infer what kind of task you're, you're making or you're doing. and then put all of these things in a database where if you recall it next time, it would remember what you were doing and what objects you're handling and so forth. so, so after learning, you know, the system, once they have enough, confidence level that I can do this sub-task autonomously,

19:58 - it'll offer the user, Hey, I can do this part. Like, if you're, if you're trying to, to make an omelet, okay, can tell you, Hey, you know, I can really get the extra treatment for you. I already learned that from you past few times. So no, go ahead, do it. Okay? But then as you progress with this, it can say, Hey, I can do the whole meal for you. You know, let me do it for you. So, so as a baby starts learning,

20:20 - the robot also starts learning and then start adapting to the person that's using that system. like if you're using the system and someone else using the system, they, they learn different things to go, to be done different ways. So that's where we're going and heading with this, project. again, interface in that case is not really the focus,

20:38 - it's the focus of, of the learning process and how you can handle different objects, without failure. the interface of course, can also be done the same way, but that we did it in the previous, previous one.

Faculti AI can make mistakes